--- title: Portworx description: Collect runtime metrics from a Portworx Instance. breadcrumbs: Docs > Integrations > Portworx --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Portworx Supported OS Integration version1.1.1 {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} ## Overview{% #overview %} Get metrics from Portworx service in real time to: - Monitor health and performance of your Portworx cluster - Track disk usage, latency, and throughput for Portworx volumes - Set up [Autopilot rules](https://docs.portworx.com/portworx-enterprise/operations/scale-portworx-cluster/autopilot) ## Setup{% #setup %} ### Installation{% #installation %} #### Create the Datadog credentials Secret{% #create-the-datadog-credentials-secret %} Create a Kubernetes Secret containing your Datadog API and application keys: ```yaml apiVersion: v1 kind: Secret metadata: name: datadog-credentials namespace: datadog-ns type: Opaque data: # Base64-encoded Datadog API Key api-key: # Base64-encoded Datadog Application Key app-key: ``` - Replace the namespace, secret name, and encoded key values as required. - Use an application key with metrics read permission for `app-key`. #### Configure the Datadog Agent to export Portworx metrics{% #configure-the-datadog-agent-to-export-portworx-metrics %} Create a Datadog Agent values file (for example, `datadog_config.yaml`): ```yaml datadog: site: "datadoghq.com" clusterName: "your-cluster-name" apiKeyExistingSecret: "datadog-credentials" kubelet: tlsVerify: false clusterChecks: enabled: true orchestratorExplorer: enabled: true clusterAgent: enabled: true admissionController: enabled: true mutateUnlabelled: false ``` Use your own `site`, `clusterName`, and `apiKeyExistingSecret`. The Agent scrapes Portworx metrics and sends them to Datadog. Install the Datadog Agent with Helm: ```shell helm repo add datadog https://helm.datadoghq.com helm upgrade --install datadog-agent datadog/datadog -f ./datadog_config.yaml ``` ### Configuration{% #configuration %} #### Annotate Portworx, Stork, and Autopilot pods{% #annotate-portworx-stork-and-autopilot-pods %} Configure Datadog Autodiscovery annotations so the Agent knows which endpoints to scrape. The typical Prometheus-style service endpoints are: | Component | Endpoint | Metrics filter | | ------------ | ------------------------------- | -------------- | | Portworx API | `http://%%host%%:17001/metrics` | `px_*` | | Stork | `http://%%host%%:9091/metrics` | `stork_*` | | Autopilot | `http://%%host%%:9628/metrics` | `autopilot_*` | **Note:** Portworx has many metrics. For most use cases, Portworx API metrics alone are sufficient. Apply a `ComponentK8sConfig` Custom Resource (CR) to add the annotations: ```yaml apiVersion: core.libopenstorage.org/v1 kind: ComponentK8sConfig metadata: name: datadog-components-config namespace: portworx spec: components: - componentNames: - Portworx API workloadConfigs: - workloadNames: - portworx-api annotations: ad.datadoghq.com/portworx-api.checks: |- { "openmetrics": { "instances": [ { "openmetrics_endpoint": "http://%%host%%:17001/metrics", "namespace": "portworx", "metrics": ["px_*"] } ] } } - componentNames: - Stork workloadConfigs: - workloadNames: - stork annotations: ad.datadoghq.com/stork.checks: |- { "openmetrics": { "instances": [ { "openmetrics_endpoint": "http://%%host%%:9091/metrics", "namespace": "portworx", "metrics": ["stork_*"] } ] } } - componentNames: - Autopilot workloadConfigs: - workloadNames: - autopilot annotations: ad.datadoghq.com/autopilot.checks: |- { "openmetrics": { "instances": [ { "openmetrics_endpoint": "http://%%host%%:9628/metrics", "namespace": "portworx", "metrics": ["autopilot_*"] } ] } } ``` At this point, the Datadog Agent is configured to scrape Portworx metrics and the Portworx, Stork, and Autopilot pods are annotated for Autodiscovery. To configure rules, see the [Autopilot documentation](https://docs.portworx.com/portworx-enterprise/operations/scale-portworx-cluster/autopilot). ### Validation{% #validation %} Run the following command to confirm the Agent is scraping Portworx metrics: ```shell kubectl exec -n -- agent status ``` Look for `openmetrics` instances under the `Checks` section with `portworx`, `stork`, and `autopilot`. ## Compatibility{% #compatibility %} This integration is compatible with Portworx 1.4.0 and later versions. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **portworx.px_backup_stats_backup_status**(gauge) | Status for this backup (0=InProgress,1=Done,2=Failed) | | **portworx.px_backup_stats_backup_size**(gauge) | Size in bytes for this backup*Shown as byte* | | **portworx.px_backup_stats_backup_duration_seconds**(gauge) | Duration in seconds for this backup*Shown as second* | | **portworx.px_backup_stats_backup_uploaded_bytes_mbs**(gauge) | Delta bytes uploaded in MB/s from last interval for this backup*Shown as byte* | | **portworx.px_backup_stats_download_size_bytes**(gauge) | Size of downloaded bytes during backup/restore from cloud*Shown as byte* | | **portworx.px_backup_stats_upload_size_bytes**(gauge) | Size of uploaded bytes during backup/restore from cloud*Shown as byte* | | **portworx.px_backup_stats_get_apis_invoked**(gauge) | Number of times GET API was invoked | | **portworx.px_backup_stats_put_apis_invoked**(gauge) | Number of times PUT API was invoked | | **portworx.px_cluster_cpu_percent**(gauge) | Percentage of CPU used*Shown as percent* | | **portworx.px_cluster_memory_utilized_percent**(gauge) | Percentage of memory utilization*Shown as percent* | | **portworx.px_cluster_disk_total_bytes**(gauge) | Total storage space in bytes for this node*Shown as byte* | | **portworx.px_cluster_disk_available_bytes**(gauge) | Available storage space in bytes for this node*Shown as byte* | | **portworx.px_cluster_disk_utilized_bytes**(gauge) | Utilized storage space in bytes for this node*Shown as byte* | | **portworx.px_cluster_pendingio**(gauge) | Number of read and write operations currently in progress for this node | | **portworx.px_cluster_trashcan_volumes**(gauge) | Number of volumes currently in the trashcan | | **portworx.px_cluster_status_cluster_size**(gauge) | Node count for your Portworx cluster. **Deprecated**. | | **portworx.px_cluster_status_size**(gauge) | Node count for your Portworx cluster | | **portworx.px_cluster_status_cluster_quorum**(gauge) | Indicates if the cluster is in quorum. **Deprecated**. | | **portworx.px_cluster_status_quorum**(gauge) | Indicates if the cluster is in quorum | | **portworx.px_cluster_status_nodes_online**(gauge) | Number of online nodes in the cluster (includes storage and storageless) | | **portworx.px_cluster_status_nodes_offline**(gauge) | Number of offline nodes in the cluster (includes storage and storageless) | | **portworx.px_cluster_status_nodes_storage_down**(gauge) | Number of nodes where the storage is full or down | | **portworx.px_cluster_status_storage_nodes_online**(gauge) | Number of storage nodes that are online | | **portworx.px_cluster_status_storage_nodes_offline**(gauge) | Number of storage nodes that are offline | | **portworx.px_cluster_status_storage_nodes_decommissioned**(gauge) | Number of storage nodes that are decommissioned | | **portworx.px_device_delete_delete_after_discard_enabled**(gauge) | This is 1 if NBDD is enabled | | **portworx.px_device_delete_discard_bytes_total**(count) | Total bytes discarded across discard requests*Shown as byte* | | **portworx.px_device_delete_discard_duration_msec_total**(count) | Total time spent for discard requests*Shown as millisecond* | | **portworx.px_device_delete_discard_errors_total**(count) | Total errors across discard requests | | **portworx.px_device_delete_discard_operations_total**(count) | Total number of discard requests | | **portworx.px_device_delete_operations_completed_total**(count) | Total number of delete operations completed | | **portworx.px_device_delete_operations_inprogress**(gauge) | Current operations in progress | | **portworx.px_device_delete_operations_queued**(gauge) | Current operations queued | | **portworx.px_disk_stats_used_bytes**(gauge) | Total storage in bytes for this disk*Shown as byte* | | **portworx.px_disk_stats_interval_seconds**(gauge) | Length of the disk stats sampling interval in seconds*Shown as second* | | **portworx.px_disk_stats_io_seconds**(gauge) | Time spent doing I/O in seconds for this disk*Shown as second* | | **portworx.px_disk_stats_progress_io**(gauge) | I/O operations currently in progress for this disk | | **portworx.px_disk_stats_disk_read_bytes**(gauge) | Total bytes read for this disk. **Deprecated**.*Shown as byte* | | **portworx.px_disk_stats_read_bytes**(gauge) | Total bytes read for this disk*Shown as byte* | | **portworx.px_disk_stats_write_bytes_seconds**(gauge) | Total written bytes for this disk. **Deprecated**.*Shown as byte* | | **portworx.px_disk_stats_written_bytes**(gauge) | Total written bytes for this disk*Shown as byte* | | **portworx.px_disk_stats_read_seconds**(gauge) | Total time spent reading in seconds for this disk*Shown as second* | | **portworx.px_disk_stats_write_seconds**(gauge) | Total time spent writing in seconds for this disk*Shown as second* | | **portworx.px_disk_stats_read_latency_seconds**(gauge) | Average time spent per read operation in seconds for this disk*Shown as second* | | **portworx.px_disk_stats_write_latency**(gauge) | Average time spent per write operation in seconds for this disk. **Deprecated**.*Shown as second* | | **portworx.px_disk_stats_write_latency_seconds**(gauge) | Average time spent per write operation in seconds for this disk*Shown as second* | | **portworx.px_disk_stats_disk_num_reads**(gauge) | Total number of read operations completed successfully for this disk. **Deprecated**. | | **portworx.px_disk_stats_disk_num_writes**(gauge) | Total number of write operations completed successfully for this disk. **Deprecated**. | | **portworx.px_disk_stats_num_reads**(gauge) | Total number of read operations completed successfully for this disk | | **portworx.px_disk_stats_num_writes**(gauge) | Total number of write operations completed successfully for this disk | | **portworx.px_disk_stats_num_reads_total**(count) | Total number of read operations completed successfully for this disk | | **portworx.px_disk_stats_num_writes_total**(count) | Total number of write operations completed successfully for this disk | | **portworx.px_disk_stats_written_bytes_total**(count) | Total bytes written for this disk*Shown as byte* | | **portworx.px_disk_stats_read_bytes_total**(count) | Total bytes read for this disk*Shown as byte* | | **portworx.px_disk_stats_read_seconds_total**(count) | Total time spent reading in seconds for this disk*Shown as second* | | **portworx.px_disk_stats_write_seconds_total**(count) | Total time spent writing in seconds for this disk*Shown as second* | | **portworx.px_fafb_apis_requests_total**(count) | Total number of API requests made to all the configured FA/FB endpoints | | **portworx.px_fafb_apis_array_requests_total**(count) | Total number of API requests made to the /arrays endpoint. | | **portworx.px_fafb_apis_login_requests_total**(count) | Total number of API requests made to the /login endpoint. | | **portworx.px_fafb_apis_controllers_requests_total**(count) | Total number of API requests made to the /controllers endpoint | | **portworx.px_fafb_apis_alerts_requests_total**(count) | Total number of API requests made to the /alerts endpoint | | **portworx.px_fafb_apis_ports_requests_total**(count) | Total number of API requests made to the /ports endpoint | | **portworx.px_fafb_apis_volumes_requests_total**(count) | Total number of API requests made to the /volumes endpoint | | **portworx.px_fafb_apis_volumesnapshots_requests_total**(count) | Total number of API requests made to the /volume-snapshots endpoint | | **portworx.px_fafb_apis_hosts_requests_total**(count) | Total number of API requests made to the /hosts endpoint | | **portworx.px_fafb_apis_connections_requests_total**(count) | Total number of API requests made to the /connections endpoint | | **portworx.px_fafb_apis_version_requests_total**(count) | Total number of API requests made to the /api_version endpoint | | **portworx.px_kvdb_get_requests_total**(count) | Total number of get requests for a key | | **portworx.px_kvdb_snapshot_requests_total**(count) | Total number of kvdb snapshot requests for a list of prefixes | | **portworx.px_kvdb_put_requests_total**(count) | Total number of put requests for a key | | **portworx.px_kvdb_create_requests_total**(count) | Total number of create requests for a key | | **portworx.px_kvdb_update_requests_total**(count) | Total number of update requests for a key | | **portworx.px_kvdb_enumerate_requests_total**(count) | Total number of enumerate requests for a key | | **portworx.px_kvdb_delete_requests_total**(count) | Total number of delete requests for a key | | **portworx.px_kvdb_keys_requests_total**(count) | Total number of list key requests for a prefix | | **portworx.px_kvdb_cas_requests_total**(count) | Total number of compare and sets for a key | | **portworx.px_kvdb_cad_requests_total**(count) | Total number of compare and deletes for a key | | **portworx.px_kvdb_lock_requests_total**(count) | Total number of lock requests for a key | | **portworx.px_kvdb_unlock_requests_total**(count) | Total number of unlock requests for a key | | **portworx.px_kvdb_watchkey_requests_total**(count) | Total number of watch requests for a key from a node | | **portworx.px_kvdb_watchtree_requests_total**(count) | Total number of watch requests for a prefix from a node | | **portworx.px_kvdb_adduser_requests_total**(count) | Total number of add user requests | | **portworx.px_kvdb_removeuser_requests_total**(count) | Total number of remove user requests | | **portworx.px_kvdb_grantuser_access_requests_total**(count) | Total number of grant access requests for a user | | **portworx.px_kvdb_revokeuser_access_requests_total**(count) | Total number of revoke access requests for a user | | **portworx.px_kvdb_addmember_requests_total**(count) | Total number of add member requests from a node | | **portworx.px_kvdb_addlearner_requests_total**(count) | Total number of add learner requests from a node | | **portworx.px_kvdb_removemember_requests_total**(count) | Total number of remove member requests from a node | | **portworx.px_kvdb_updatemember_requests_total**(count) | Total number of update member requests from a node | | **portworx.px_kvdb_listmembers_requests_total**(count) | Total number of list member requests from a node | | **portworx.px_kvdb_setendpoints_requests_total**(count) | Total number of set endpoint requests from a node | | **portworx.px_kvdb_getendpoints_requests_total**(count) | Total number of get endpoint requests from a node | | **portworx.px_kvdb_defragment_requests_total**(count) | Total number of defragment requests from a node | | **portworx.px_kvdb_watch_latency_seconds**(gauge) | Time taken in seconds between a kvdb put and a corresponding watch update*Shown as second* | | **portworx.px_kvdb_health_state_node_view**(gauge) | This node's kvdb state (1 = healthy, 2 = not healthy) | | **portworx.px_kvdb_health_state_cluster_view**(gauge) | This node's view of other node's kvdb state (1 = healthy, 2 = not healthy) | | **portworx.px_network_io_bytessent**(gauge) | Number of bytes sent during this interval*Shown as byte* | | **portworx.px_network_io_received_bytes**(gauge) | Number of bytes received during this interval*Shown as byte* | | **portworx.px_network_io_sent_bytes_total**(count) | Total number of bytes sent*Shown as byte* | | **portworx.px_network_io_received_bytes_total**(count) | Total number of bytes received*Shown as byte* | | **portworx.px_network_io_reservation_waits**(gauge) | Total number of SM waits for reservation | | **portworx.px_network_io_reservation_wake_ups**(gauge) | Total number of SM wake ups after getting the reservation | | **portworx.px_nfs_pool_stats_packets_arrived**(gauge) | Counts the number of NFS packets arrived on the socket. | | **portworx.px_nfs_pool_stats_sockets_enqueued**(gauge) | Counts how many times an NFS transport is enqueued to wait for an nfsd thread to service it, i.e. no nfsd thread was considered available. | | **portworx.px_nfs_pool_stats_threads_woken**(gauge) | Counts how many times an idle nfsd thread is woken to try to receive some data from an NFS transport. | | **portworx.px_nfs_pool_stats_threads_timedout**(gauge) | Counts how many times an nfsd thread triggered an idle timeout, i.e. was not woken to handle any incoming network packets for some time. | | **portworx.px_node_stats_used_mem**(gauge) | Used memory in bytes*Shown as byte* | | **portworx.px_node_stats_cpu_usage**(gauge) | Percent of CPU consumption*Shown as percent* | | **portworx.px_node_stats_busy_threads_percent**(gauge) | Percentage of Busy Threads*Shown as percent* | | **portworx.px_node_stats_thread_pool_high_priority_queue_size**(gauge) | Thread Pool High Priority Queue Size | | **portworx.px_node_stats_writeback_list_num_dirty_pools**(gauge) | Writeback List Number of Dirty Pools | | **portworx.px_node_stats_writeback_list_num_pending_log_bufs**(gauge) | Writeback List Number of Pending Log Buffers | | **portworx.px_node_stats_writeback_list_low_cos_queue_size**(gauge) | Writeback List Low Cos Queue Size | | **portworx.px_node_stats_writeback_list_high_cos_queue_size**(gauge) | Writeback List High Cos Queue Size | | **portworx.px_node_stats_sm_run_queue_size**(gauge) | Statemachine Run Queue Size | | **portworx.px_node_stats_kaio_pending_queue_size**(gauge) | Kaio Pending Queue Size | | **portworx.px_node_stats_request_scheduler_queue_size**(gauge) | Request Scheduler Queue Size | | **portworx.px_node_stats_procfs_mem_available_bytes**(gauge) | Available memory in bytes*Shown as byte* | | **portworx.px_node_stats_procfs_mem_dirty_bytes**(gauge) | The total amount of memory waiting to be written back to the disk*Shown as byte* | | **portworx.px_node_stats_procfs_mem_writeback_bytes**(gauge) | The total amount of memory actively being written back to the disk*Shown as byte* | | **portworx.px_node_stats_procfs_mem_total_bytes**(gauge) | Total amount of usable RAM which is physical RAM minus a number of reserved bits and the kernel binary code*Shown as byte* | | **portworx.px_node_stats_procfs_mem_free_bytes**(gauge) | The amount of physical RAM, in bytes, left unused by the system.*Shown as byte* | | **portworx.px_node_stats_procfs_mem_buffers_bytes**(gauge) | The amount, in bytes, of temporary storage for raw disk blocks*Shown as byte* | | **portworx.px_node_stats_procfs_mem_cached_bytes**(gauge) | The amount of physical RAM, in bytes, used as cache memory*Shown as byte* | | **portworx.px_node_stats_procfs_mem_active_anon_bytes**(gauge) | The amount of anonymous and tmpfs/shmem memory, in bytes, that is in active use*Shown as byte* | | **portworx.px_node_stats_procfs_mem_inactive_anon_bytes**(gauge) | The amount of anonymous and tmpfs/shmem memory, in bytes, that is a candidate for eviction*Shown as byte* | | **portworx.px_node_stats_procfs_mem_active_file_bytes**(gauge) | The amount of file cache memory, in bytes, that is in active use*Shown as byte* | | **portworx.px_node_stats_procfs_mem_inactive_file_bytes**(gauge) | The amount of file cache memory, in bytes, that is newly loaded from the disk*Shown as byte* | | **portworx.px_node_stats_procfs_mem_unevictable_bytes**(gauge) | The amount of memory, in bytes, discovered by the pageout code*Shown as byte* | | **portworx.px_node_stats_procfs_mem_mlocked_bytes**(gauge) | The total amount of memory, in bytes, that is not evictable because it is locked into memory by user programs*Shown as byte* | | **portworx.px_node_stats_procfs_mem_anon_pages_bytes**(gauge) | The total amount of memory, in bytes, used by pages that are not backed by files and are mapped into userspace page tables*Shown as byte* | | **portworx.px_node_stats_procfs_mem_mapped_bytes**(gauge) | The memory, in bytes, used for files that have been mmaped*Shown as byte* | | **portworx.px_node_stats_procfs_mem_sh_mem_bytes**(gauge) | The total amount of memory, in bytes, used by shared memory (shmem) and tmpfs*Shown as byte* | | **portworx.px_node_stats_num_skinnysnaps**(gauge) | The number of SkinnySnaps | | **portworx.px_node_stats_skinnysnaps_num_repls_skips**(gauge) | The number of target replicas skipped for a snapshot due to SkinnySnap | | **portworx.px_node_stats_skinnysnaps_num_repls_snapshots**(gauge) | The number of target replicas that underwent a snapshot operation due to SkinnySnap | | **portworx.px_node_stats_relaxed_reclaim_pending**(gauge) | The number of volumes in RelaxedReclaim pending queue | | **portworx.px_node_stats_relaxed_reclaim_skipped**(gauge) | The number of volume deletes that were skipped from being delayed even when RelaxedReclaim was enabled. | | **portworx.px_node_stats_relaxed_reclaim_deleted**(gauge) | The number of volume deletes done through RelaxedReclaim | | **portworx.px_node_stats_num_volumes**(gauge) | The number of volumes hosted on this node | | **portworx.px_node_status_node_status**(gauge) | Status of this node ([https://libopenstorage.github.io/w/master.generated-api/#status](https://libopenstorage.github.io/w/master.generated-api/#status)) | | **portworx.px_node_status_license_expiry**(gauge) | Number of days until License (or License lease) expires (<0 means Expired) | | **portworx.px_none_stats_free_mem**(gauge) | Available memory in bytes*Shown as byte* | | **portworx.px_none_stats_total_mem**(gauge) | Total memory in bytes*Shown as byte* | | **portworx.px_pool_stats_pool_written_bytes**(gauge) | Bytes written since last interval for this pool. **Deprecated**.*Shown as byte* | | **portworx.px_pool_stats_pool_write_latency_seconds**(gauge) | Average time spent per write operation for this pool. **Deprecated**.*Shown as second* | | **portworx.px_pool_stats_pool_writethroughput**(gauge) | Average number of bytes written per second for this pool. **Deprecated**.*Shown as byte* | | **portworx.px_pool_stats_pool_flushed_bytes**(gauge) | Number of flushed bytes since last interval for this pool. **Deprecated**.*Shown as byte* | | **portworx.px_pool_stats_pool_num_flushes**(gauge) | Number of flush(sync) operations since last interval for this pool. **Deprecated**. | | **portworx.px_pool_stats_pool_flushms**(gauge) | Latency for flush for this pool. **Deprecated**.*Shown as millisecond* | | **portworx.px_pool_stats_pool_provisioned_bytes**(gauge) | Provisioned storage space in bytes for this pool. **Deprecated**.*Shown as byte* | | **portworx.px_pool_stats_pool_status**(gauge) | Status of this Pool (0=Offline,1=Online). **Deprecated**. | | **portworx.px_pool_stats_written_bytes**(gauge) | Bytes written since last interval for this pool*Shown as byte* | | **portworx.px_pool_stats_num_writes**(gauge) | Number of write operations in the last interval for this pool | | **portworx.px_pool_stats_write_ms**(gauge) | Total time in millisecond spent in writing in the last interval for this pool*Shown as millisecond* | | **portworx.px_pool_stats_write_latency_seconds**(gauge) | Average time spent per write operation for this pool*Shown as second* | | **portworx.px_pool_stats_write_iops**(gauge) | Average number of completed write operations per second for this pool | | **portworx.px_pool_stats_writethroughput**(gauge) | Average number of bytes written per second for this pool*Shown as byte* | | **portworx.px_pool_stats_flushed_bytes**(gauge) | Number of flushed bytes since last interval for this pool*Shown as byte* | | **portworx.px_pool_stats_num_flushes**(gauge) | Number of flush(sync) operations since last interval for this pool | | **portworx.px_pool_stats_flushms**(gauge) | Latency for flush for this pool*Shown as millisecond* | | **portworx.px_pool_stats_provisioned_bytes**(gauge) | Provisioned storage space in bytes for this pool*Shown as byte* | | **portworx.px_pool_stats_status**(gauge) | Status of this Pool (0=Offline, 1=Online, 2=Full, 3=NotFound, 4=Maintenance) | | **portworx.px_pool_stats_available_bytes**(gauge) | Available storage space in bytes for this pool*Shown as byte* | | **portworx.px_pool_stats_used_bytes**(gauge) | Used storage space in bytes for this pool*Shown as byte* | | **portworx.px_pool_stats_total_bytes**(gauge) | Total storage space in bytes for this pool*Shown as byte* | | **portworx.px_pool_stats_written_bytes_total**(count) | Total bytes written for this pool*Shown as byte* | | **portworx.px_pool_stats_flushed_bytes_total**(count) | Total number of flushed bytes*Shown as byte* | | **portworx.px_pool_stats_num_flushes_total**(count) | Total number of flush(sync) operations | | **portworx.px_pool_stats_flushms_total**(count) | Total time spent in flush*Shown as millisecond* | | **portworx.px_proc_stats_virt**(gauge) | Virtual memory in bytes*Shown as byte* | | **portworx.px_proc_stats_res**(gauge) | Resident set size memory in bytes*Shown as byte* | | **portworx.px_proc_stats_cputime**(gauge) | Amount of time that this process has been scheduled in user and kernel mode measured in clock ticks | | **portworx.px_px_cache_status**(gauge) | Cache enabled (0=No,1=Yes) | | **portworx.px_px_cache_total_blocks**(gauge) | Number of total blocks in the cache | | **portworx.px_px_cache_used_blocks**(gauge) | Number of used blocks in the cache | | **portworx.px_px_cache_dirty_blocks**(gauge) | Number of dirty blocks in the cache | | **portworx.px_px_cache_read_hits**(gauge) | Number of read hits for the cache | | **portworx.px_px_cache_read_miss**(gauge) | Number of read misses for the cache | | **portworx.px_px_cache_write_hits**(gauge) | Number of write hits for the cache | | **portworx.px_px_cache_write_miss**(gauge) | Number of write misses for the cache | | **portworx.px_px_cache_block_size**(gauge) | Block size for the cache | | **portworx.px_px_cache_mode**(gauge) | Mode of the cache | | **portworx.px_px_cache_migrate_promote**(gauge) | Number of blocks promoted to the cache | | **portworx.px_px_cache_migrate_demote**(gauge) | Number of block demoted from the cache | | **portworx.px_px_cache_io_mbps**(gauge) | Approximate cache bandwidth from cache internal actions | | **portworx.px_rebalance_rebalance_job_state**(gauge) | Rebalance job state (0 = pending, 1 = running, 2 = done, 3 = paused, 4 = cancelled) | | **portworx.px_rebalance_provision_space_rebalanced_bytes_total**(count) | Total provisioned space rebalanced (only counts add (since remove has equal value as add))*Shown as byte* | | **portworx.px_rebalance_used_space_rebalanced_bytes_total**(count) | Total used space rebalanced (only counts add (since remove has equal value as add))*Shown as byte* | | **portworx.px_rebalance_volumes_rebalanced_total**(count) | Total volumes affected by rebalance operation | | **portworx.px_rebalance_overloaded_pools_total**(count) | Number of overloaded pools | | **portworx.px_volume_usage_bytes**(gauge) | Used storage space in bytes for this volume*Shown as byte* | | **portworx.px_volume_capacity_bytes**(gauge) | Configured size in bytes for this volume*Shown as byte* | | **portworx.px_volume_halevel**(gauge) | Configured HA level for this volume | | **portworx.px_volume_currhalevel**(gauge) | Current HA level for this volume | | **portworx.px_volume_iopriority**(gauge) | Configured I/O priority for this volume | | **portworx.px_volume_elapsed_time_since_detached_seconds**(gauge) | Seconds elapsed since the volume is detached*Shown as second* | | **portworx.px_volume_elapsed_time_since_attached_seconds**(gauge) | Seconds elapsed since the volume is attached*Shown as second* | | **portworx.px_volume_attached**(gauge) | Attached state for this volume (0=detached,1=attached) | | **portworx.px_volume_status**(gauge) | Status for this volume ([https://libopenstorage.github.io/w/master.generated-api/#volumestatus](https://libopenstorage.github.io/w/master.generated-api/#volumestatus)) | | **portworx.px_volume_state**(gauge) | State for this volume ([https://libopenstorage.github.io/w/master.generated-api/#volumestate](https://libopenstorage.github.io/w/master.generated-api/#volumestate)) | | **portworx.px_volume_attached_state**(gauge) | Attached state for this volume (valid only if volume is attached) ([https://libopenstorage.github.io/w/master.generated-api/#attachstate](https://libopenstorage.github.io/w/master.generated-api/#attachstate)) | | **portworx.px_volume_fs_health_status**(gauge) | Filesystem health status for this volume ([https://libopenstorage.github.io/w/master.generated-api/#filesystemhealthstatus](https://libopenstorage.github.io/w/master.generated-api/#filesystemhealthstatus)) | | **portworx.px_volume_replication_status**(gauge) | Replication Status for this volume (0 : up, 1 : not in quorum, 2 : resync state, 3 : degraded, 4 : detached, 5 : restore) | | **portworx.px_volume_fs_usage_bytes**(gauge) | Used storage space in bytes as reported by the filesystem for this volume*Shown as byte* | | **portworx.px_volume_fs_capacity_bytes**(gauge) | Total storage space in bytes as reported by the filesystem for this volume*Shown as byte* | | **portworx.px_volume_vol_read_bytes**(gauge) | Number of successfully read bytes during this interval for this volume. **Deprecated**.*Shown as byte* | | **portworx.px_volume_vol_written_bytes**(gauge) | Number of successfully written bytes during this interval for this volume. **Deprecated**.*Shown as byte* | | **portworx.px_volume_vol_reads**(gauge) | Number of successfully completed read operations during this interval for this volume. **Deprecated**. | | **portworx.px_volume_vol_writes**(gauge) | Number of successfully completed write operations during this interval for this volume. **Deprecated**. | | **portworx.px_volume_read_bytes**(gauge) | Number of successfully read bytes during this interval for this volume*Shown as byte* | | **portworx.px_volume_written_bytes**(gauge) | Number of successfully written bytes during this interval for this volume*Shown as byte* | | **portworx.px_volume_reads**(gauge) | Number of successfully completed read operations during this interval for this volume | | **portworx.px_volume_writes**(gauge) | Number of successfully completed write operations during this interval for this volume | | **portworx.px_volume_read_bytes_total**(count) | Total number of successfully read bytes for this volume*Shown as byte* | | **portworx.px_volume_written_bytes_total**(count) | Total number of successfully written bytes for this volume*Shown as byte* | | **portworx.px_volume_reads_total**(count) | Total number of successfully completed read operations for this volume | | **portworx.px_volume_writes_total**(count) | Total number of successfully completed write operations for this volume | | **portworx.px_volume_iops**(gauge) | Number of successful completed I/O operations per second during this interval for this volume | | **portworx.px_volume_write_iops**(gauge) | Average number of completed write operations per second for this volume | | **portworx.px_volume_read_iops**(gauge) | Average number of completed read operations per second for this volume | | **portworx.px_volume_vol_num_sequential_writes**(gauge) | Number of sequential write I/O operations during this interval for this volume | | **portworx.px_volume_vol_num_sequential_reads**(gauge) | Number of sequential read I/O operations during this interval for this volume | | **portworx.px_volume_vol_num_random_writes**(gauge) | Number of random write I/O operations during this interval for this volume | | **portworx.px_volume_vol_num_random_reads**(gauge) | Number of random read I/O operations during this interval for this volume | | **portworx.px_volume_depth_io**(gauge) | Number of I/O operations currently in progress for this volume | | **portworx.px_volume_readthroughput**(gauge) | Number of bytes read per second during this interval for this volume*Shown as byte* | | **portworx.px_volume_writethroughput**(gauge) | Number of bytes written per second during this interval for this volume*Shown as byte* | | **portworx.px_volume_vol_bytes_reclaimed**(gauge) | Number of bytes reclaimed by fstrim operation*Shown as byte* | | **portworx.px_volume_vol_bytes_reclaimable**(gauge) | Number of bytes reclaimable on the volume as seen by fstrim operation*Shown as byte* | | **portworx.px_volume_vol_read_latency_seconds**(gauge) | Average time spent per successfully completed read operation in seconds during this interval for this volume. **Deprecated**.*Shown as second* | | **portworx.px_volume_vol_write_latency_seconds**(gauge) | Average time spent per successfully completed write operation in seconds during this interval for this volume. **Deprecated**.*Shown as second* | | **portworx.px_volume_read_latency_seconds**(gauge) | Average time spent per successfully completed read operation in seconds for this volume*Shown as second* | | **portworx.px_volume_write_latency_seconds**(gauge) | Average time spent per successfully completed write operation in seconds for this volume*Shown as second* | | **portworx.px_volume_num_long_reads**(gauge) | Number of long reads for this volume | | **portworx.px_volume_num_long_writes**(gauge) | Number of long writes for this volume | | **portworx.px_volume_num_long_flushes**(gauge) | Number of long flushes for this volume | | **portworx.px_volume_num_db_flushes**(gauge) | Number of DB flushes for this volume | | **portworx.px_volume_replica_read_bytes_total**(count) | Total number of successfully read bytes for this replica volume*Shown as byte* | | **portworx.px_volume_replica_written_bytes_total**(count) | Total number of successfully written bytes for this replica volume*Shown as byte* | | **portworx.px_volume_replica_reads_total**(count) | Total number of successfully completed read operations for this replica volume | | **portworx.px_volume_replica_writes_total**(count) | Total number of successfully completed write operations for this replica volume | | **portworx.px_volume_replica_flushes_total**(count) | Total number of successfully completed flush operations for this replica volume | | **portworx.px_volume_replica_read_ms_total**(count) | Total time spent doing read operations in milliseconds for this replica volume*Shown as millisecond* | | **portworx.px_volume_replica_write_ms_total**(count) | Total time doing write operations in milliseconds for this replica volume*Shown as millisecond* | | **portworx.px_volume_replica_flush_ms_total**(count) | Total time doing write operations in milliseconds for this replica volume*Shown as millisecond* | | **portworx.px_volume_dev_depth_io**(gauge) | Number of I/O operations currently in progress as reported by the kernel pxd device for this volume | | **portworx.px_volume_dev_writethroughput**(gauge) | Number of successfully written bytes per second as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px_volume_dev_readthroughput**(gauge) | Number of successfully read bytes per second as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px_volume_dev_read_latency_seconds**(gauge) | Average time spent per successfully completed read in seconds as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px_volume_dev_write_latency_seconds**(gauge) | Average time spent per successfully completed write in seconds as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px_volume_dev_read_bytes_total**(count) | Total number of successfully read bytes as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px_volume_dev_written_bytes_total**(count) | Number of successfully written bytes as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px_volume_dev_reads_total**(count) | Total number of successfully completed read operations as reported by the kernel pxd device for this volume | | **portworx.px_volume_dev_writes_total**(count) | Total number of successfully completed write operations as reported by the kernel pxd device for this volume | | **portworx.px_volume_dev_read_seconds_total**(count) | Total time spent reading in seconds for this disk as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px_volume_dev_write_seconds_total**(count) | Total time spent writing in seconds for this disk as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px_volume_unique_blocks**(gauge) | Size(in bytes) of unique blocks for this volume*Shown as byte* | | **portworx.px_volume_timestamp_records**(gauge) | Number of timestamp records accumulated | | **portworx.px_volume_timestamp_records_per_node**(gauge) | Number of timestamp records accumulated for a node | | **portworx.px_volume_num_discard_ops**(gauge) | Number of discard operations for this volume | | **portworx.px_volume_discarded_bytes**(gauge) | Number of discarded bytes for this volume*Shown as byte* | | **portworx.autopilot_errors_total**(count) | Total number of autopilot errors | | **portworx.autopilot_action_failed_count**(gauge) | Total number of declined or failed actions | | **portworx.autopilot_action_failed_total**(count) | Total number of failed actions per rule object | | **portworx.autopilot_objects_in_stable_condition_total**(count) | Total number of objects with stable conditions per rule | | **portworx.autopilot_action_taken_total**(count) | Total number of actions taken per rule object | | **portworx.autopilot_objects_total**(count) | Total number of objects per rule | | **portworx.autopilot_objects_with_rules_total**(count) | Total number of objects matching any rules | | **portworx.autopilot_rules_in_system**(gauge) | Total number of rules in the system | | **portworx.autopilot_dynamic_throttle_applied_total**(count) | Total number of dynamic throttles applied | | **portworx.autopilot_dynamic_throttle_failed_total**(count) | Total number of dynamic throttles failed to apply | | **portworx.autopilot_dynamic_throttle_skipped_total**(count) | Total number of dynamic throttles skipped to apply | | **portworx.autopilot_datadog_success_total**(count) | Total number of successful Datadog queries | | **portworx.autopilot_datadog_failures_total**(count) | Total number of failed Datadog queries | | **portworx.autopilot_datadog_query_validation_failures_total**(count) | Total number of failed Datadog query validations | | **portworx.autopilot_datadog_empty_response_total**(count) | Total number of empty Datadog responses | | **portworx.autopilot_prometheus_empty_response_total**(count) | Total number of empty Prometheus responses | | **portworx.stork_migration_status**(gauge) | Migration status - used to filter and list migrations by status within the cluster. | | **portworx.stork_migration_stage**(gauge) | Stage of migration (0 = Initial, 1 = PreExecRule, 2 = PostExecRule, 3 = Volumes, 4 = Applications, 5 = Final). | | **portworx.stork_migration_duration**(gauge) | Duration of migration in seconds. Stage label values: `volume` (volume migration) or `resource` (resource migration).*Shown as second* | | **portworx.stork_migration_duration_seconds**(gauge) | Total time taken by a migration in seconds.*Shown as second* | | **portworx.stork_migration_resources**(gauge) | Count of resources collected for migration (state: planned or migrated). | | **portworx.stork_migration_volumes**(gauge) | Volumes planned for migration (state: planned or migrated). | | **portworx.stork_migration_created_total**(count) | Increments when a migration is created. | | **portworx.stork_migration_completed_total**(count) | Increments when a migration reaches a terminal state (Successful, Failed, or PartialSuccess). | | **portworx.stork_migration_transfer_rate**(gauge) | Data transfer rate for a single migration (bytes/second).*Shown as byte* | | **portworx.stork_migration_transferred_bytes**(gauge) | Total bytes transferred during a single migration.*Shown as byte* | | **portworx.stork_migration_completion_timestamp_seconds**(gauge) | Timestamp when the latest migration completed for a schedule with the status being Successful, PartialSuccess, or Failed.*Shown as second* | | **portworx.stork_migration_schedule_status**(gauge) | Status of migration schedules. | | **portworx.stork_migration_schedule_last_run_status**(gauge) | Status of the latest run of each migration schedule. | | **portworx.stork_action_status**(gauge) | Status of Failover or Failback actions. | | **portworx.stork_action_created_total**(count) | Increments when a Failover/Failback action is created. | | **portworx.stork_action_completed_total**(count) | Increments when a Failover/Failback action reaches a terminal state. | | **portworx.stork_action_completion_timestamp_seconds**(gauge) | Timestamp when a Failover/Failback action completes with the status as Successful, Failed, RollbackSuccessful, or RollbackFailed.*Shown as second* | | **portworx.stork_clusterpair_scheduler_status**(gauge) | Status of the cluster pair's health with respect to Kubernetes (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). | | **portworx.stork_clusterpair_storage_status**(gauge) | Status of the cluster pair's health with respect to storage provisioner (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). | | **portworx.stork_all_namespaces**(gauge) | Number of namespaces in the Kubernetes cluster, including Kubernetes system namespaces. | | **portworx.stork_protected_namespaces**(gauge) | Count of namespaces being migrated through active migration schedules. | | **portworx.stork_hyperconverged_pods_total**(gauge) | Number of pods scheduled on a node where replicas for all pod volumes exist (hyper-converged by the Stork scheduler). | | **portworx.stork_non_hyperconverged_pods_total**(gauge) | Number of pods not hyper-converged by the Stork scheduler. | | **portworx.stork_semi_hyperconverged_pods_total**(gauge) | Number of pods partially hyper-converged (scheduled on nodes with some but not all volume replicas). | ### Events{% #events %} The Portworx integration does not include any events. ### Service Checks{% #service-checks %} The Portworx integration does not include any service checks. ## Troubleshooting{% #troubleshooting %} ### Agent cannot scrape metrics{% #agent-cannot-scrape-metrics %} If metrics are not appearing in Datadog, check the following: - Verify the `ComponentK8sConfig` CR was applied and the pods have the expected annotations: ```shell kubectl get pod -n portworx -o jsonpath='{.metadata.annotations}' ``` - Confirm the OpenMetrics endpoints are reachable from within the Agent pod: ```shell kubectl exec -n -- curl http://:17001/metrics ``` - Check Agent logs for scrape errors: ```shell kubectl logs -n ``` ## Support{% #support %} Need help? Contact [Portworx support](mailto:support@purestorage.com). ## Further Reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Portworx + Datadog: Monitor your complete container infrastructure](https://www.datadoghq.com/blog/portworx-integration/)