--- title: Portworx description: Collect runtime metrics from a Portworx Instance. breadcrumbs: Docs > Integrations > Portworx --- # Portworx Supported OS Integration version1.1.1 {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} ## Overview{% #overview %} Get metrics from Portworx service in real time to: - Monitor health and performance of your Portworx cluster - Track disk usage, latency, and throughput for Portworx volumes - Set up [Autopilot rules](https://docs.portworx.com/portworx-enterprise/operations/scale-portworx-cluster/autopilot) ## Setup{% #setup %} ### Installation{% #installation %} #### Create the Datadog credentials Secret{% #create-the-datadog-credentials-secret %} Create a Kubernetes Secret containing your Datadog API and application keys: ```yaml apiVersion: v1 kind: Secret metadata: name: datadog-credentials namespace: datadog-ns type: Opaque data: # Base64-encoded Datadog API Key api-key: # Base64-encoded Datadog Application Key app-key: ``` - Replace the namespace, secret name, and encoded key values as required. - Use an application key with metrics read permission for `app-key`. #### Configure the Datadog Agent to export Portworx metrics{% #configure-the-datadog-agent-to-export-portworx-metrics %} Create a Datadog Agent values file (for example, `datadog_config.yaml`): ```yaml datadog: site: "datadoghq.com" clusterName: "your-cluster-name" apiKeyExistingSecret: "datadog-credentials" kubelet: tlsVerify: false clusterChecks: enabled: true orchestratorExplorer: enabled: true clusterAgent: enabled: true admissionController: enabled: true mutateUnlabelled: false ``` Use your own `site`, `clusterName`, and `apiKeyExistingSecret`. The Agent scrapes Portworx metrics and sends them to Datadog. Install the Datadog Agent with Helm: ```shell helm repo add datadog https://helm.datadoghq.com helm upgrade --install datadog-agent datadog/datadog -f ./datadog_config.yaml ``` ### Configuration{% #configuration %} #### Annotate Portworx, Stork, and Autopilot pods{% #annotate-portworx-stork-and-autopilot-pods %} Configure Datadog Autodiscovery annotations so the Agent knows which endpoints to scrape. The typical Prometheus-style service endpoints are: | Component | Endpoint | Metrics filter | | ------------ | ------------------------------- | -------------- | | Portworx API | `http://%%host%%:17001/metrics` | `px_*` | | Stork | `http://%%host%%:9091/metrics` | `stork_*` | | Autopilot | `http://%%host%%:9628/metrics` | `autopilot_*` | **Note:** Portworx has many metrics. For most use cases, Portworx API metrics alone are sufficient. Apply a `ComponentK8sConfig` Custom Resource (CR) to add the annotations: ```yaml apiVersion: core.libopenstorage.org/v1 kind: ComponentK8sConfig metadata: name: datadog-components-config namespace: portworx spec: components: - componentNames: - Portworx API workloadConfigs: - workloadNames: - portworx-api annotations: ad.datadoghq.com/portworx-api.checks: |- { "openmetrics": { "instances": [ { "openmetrics_endpoint": "http://%%host%%:17001/metrics", "namespace": "portworx", "metrics": ["px_*"] } ] } } - componentNames: - Stork workloadConfigs: - workloadNames: - stork annotations: ad.datadoghq.com/stork.checks: |- { "openmetrics": { "instances": [ { "openmetrics_endpoint": "http://%%host%%:9091/metrics", "namespace": "portworx", "metrics": ["stork_*"] } ] } } - componentNames: - Autopilot workloadConfigs: - workloadNames: - autopilot annotations: ad.datadoghq.com/autopilot.checks: |- { "openmetrics": { "instances": [ { "openmetrics_endpoint": "http://%%host%%:9628/metrics", "namespace": "portworx", "metrics": ["autopilot_*"] } ] } } ``` At this point, the Datadog Agent is configured to scrape Portworx metrics and the Portworx, Stork, and Autopilot pods are annotated for Autodiscovery. To configure rules, see the [Autopilot documentation](https://docs.portworx.com/portworx-enterprise/operations/scale-portworx-cluster/autopilot). ### Validation{% #validation %} Run the following command to confirm the Agent is scraping Portworx metrics: ```shell kubectl exec -n -- agent status ``` Look for `openmetrics` instances under the `Checks` section with `portworx`, `stork`, and `autopilot`. ## Compatibility{% #compatibility %} This integration is compatible with Portworx 1.4.0 and later versions. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **portworx.px\_backup\_stats\_backup\_status**(gauge) | Status for this backup (0=InProgress,1=Done,2=Failed) | | **portworx.px\_backup\_stats\_backup\_size**(gauge) | Size in bytes for this backup*Shown as byte* | | **portworx.px\_backup\_stats\_backup\_duration\_seconds**(gauge) | Duration in seconds for this backup*Shown as second* | | **portworx.px\_backup\_stats\_backup\_uploaded\_bytes\_mbs**(gauge) | Delta bytes uploaded in MB/s from last interval for this backup*Shown as byte* | | **portworx.px\_backup\_stats\_download\_size\_bytes**(gauge) | Size of downloaded bytes during backup/restore from cloud*Shown as byte* | | **portworx.px\_backup\_stats\_upload\_size\_bytes**(gauge) | Size of uploaded bytes during backup/restore from cloud*Shown as byte* | | **portworx.px\_backup\_stats\_get\_apis\_invoked**(gauge) | Number of times GET API was invoked | | **portworx.px\_backup\_stats\_put\_apis\_invoked**(gauge) | Number of times PUT API was invoked | | **portworx.px\_cluster\_cpu\_percent**(gauge) | Percentage of CPU used*Shown as percent* | | **portworx.px\_cluster\_memory\_utilized\_percent**(gauge) | Percentage of memory utilization*Shown as percent* | | **portworx.px\_cluster\_disk\_total\_bytes**(gauge) | Total storage space in bytes for this node*Shown as byte* | | **portworx.px\_cluster\_disk\_available\_bytes**(gauge) | Available storage space in bytes for this node*Shown as byte* | | **portworx.px\_cluster\_disk\_utilized\_bytes**(gauge) | Utilized storage space in bytes for this node*Shown as byte* | | **portworx.px\_cluster\_pendingio**(gauge) | Number of read and write operations currently in progress for this node | | **portworx.px\_cluster\_trashcan\_volumes**(gauge) | Number of volumes currently in the trashcan | | **portworx.px\_cluster\_status\_cluster\_size**(gauge) | Node count for your Portworx cluster. **Deprecated**. | | **portworx.px\_cluster\_status\_size**(gauge) | Node count for your Portworx cluster | | **portworx.px\_cluster\_status\_cluster\_quorum**(gauge) | Indicates if the cluster is in quorum. **Deprecated**. | | **portworx.px\_cluster\_status\_quorum**(gauge) | Indicates if the cluster is in quorum | | **portworx.px\_cluster\_status\_nodes\_online**(gauge) | Number of online nodes in the cluster (includes storage and storageless) | | **portworx.px\_cluster\_status\_nodes\_offline**(gauge) | Number of offline nodes in the cluster (includes storage and storageless) | | **portworx.px\_cluster\_status\_nodes\_storage\_down**(gauge) | Number of nodes where the storage is full or down | | **portworx.px\_cluster\_status\_storage\_nodes\_online**(gauge) | Number of storage nodes that are online | | **portworx.px\_cluster\_status\_storage\_nodes\_offline**(gauge) | Number of storage nodes that are offline | | **portworx.px\_cluster\_status\_storage\_nodes\_decommissioned**(gauge) | Number of storage nodes that are decommissioned | | **portworx.px\_device\_delete\_delete\_after\_discard\_enabled**(gauge) | This is 1 if NBDD is enabled | | **portworx.px\_device\_delete\_discard\_bytes\_total**(count) | Total bytes discarded across discard requests*Shown as byte* | | **portworx.px\_device\_delete\_discard\_duration\_msec\_total**(count) | Total time spent for discard requests*Shown as millisecond* | | **portworx.px\_device\_delete\_discard\_errors\_total**(count) | Total errors across discard requests | | **portworx.px\_device\_delete\_discard\_operations\_total**(count) | Total number of discard requests | | **portworx.px\_device\_delete\_operations\_completed\_total**(count) | Total number of delete operations completed | | **portworx.px\_device\_delete\_operations\_inprogress**(gauge) | Current operations in progress | | **portworx.px\_device\_delete\_operations\_queued**(gauge) | Current operations queued | | **portworx.px\_disk\_stats\_used\_bytes**(gauge) | Total storage in bytes for this disk*Shown as byte* | | **portworx.px\_disk\_stats\_interval\_seconds**(gauge) | Length of the disk stats sampling interval in seconds*Shown as second* | | **portworx.px\_disk\_stats\_io\_seconds**(gauge) | Time spent doing I/O in seconds for this disk*Shown as second* | | **portworx.px\_disk\_stats\_progress\_io**(gauge) | I/O operations currently in progress for this disk | | **portworx.px\_disk\_stats\_disk\_read\_bytes**(gauge) | Total bytes read for this disk. **Deprecated**.*Shown as byte* | | **portworx.px\_disk\_stats\_read\_bytes**(gauge) | Total bytes read for this disk*Shown as byte* | | **portworx.px\_disk\_stats\_write\_bytes\_seconds**(gauge) | Total written bytes for this disk. **Deprecated**.*Shown as byte* | | **portworx.px\_disk\_stats\_written\_bytes**(gauge) | Total written bytes for this disk*Shown as byte* | | **portworx.px\_disk\_stats\_read\_seconds**(gauge) | Total time spent reading in seconds for this disk*Shown as second* | | **portworx.px\_disk\_stats\_write\_seconds**(gauge) | Total time spent writing in seconds for this disk*Shown as second* | | **portworx.px\_disk\_stats\_read\_latency\_seconds**(gauge) | Average time spent per read operation in seconds for this disk*Shown as second* | | **portworx.px\_disk\_stats\_write\_latency**(gauge) | Average time spent per write operation in seconds for this disk. **Deprecated**.*Shown as second* | | **portworx.px\_disk\_stats\_write\_latency\_seconds**(gauge) | Average time spent per write operation in seconds for this disk*Shown as second* | | **portworx.px\_disk\_stats\_disk\_num\_reads**(gauge) | Total number of read operations completed successfully for this disk. **Deprecated**. | | **portworx.px\_disk\_stats\_disk\_num\_writes**(gauge) | Total number of write operations completed successfully for this disk. **Deprecated**. | | **portworx.px\_disk\_stats\_num\_reads**(gauge) | Total number of read operations completed successfully for this disk | | **portworx.px\_disk\_stats\_num\_writes**(gauge) | Total number of write operations completed successfully for this disk | | **portworx.px\_disk\_stats\_num\_reads\_total**(count) | Total number of read operations completed successfully for this disk | | **portworx.px\_disk\_stats\_num\_writes\_total**(count) | Total number of write operations completed successfully for this disk | | **portworx.px\_disk\_stats\_written\_bytes\_total**(count) | Total bytes written for this disk*Shown as byte* | | **portworx.px\_disk\_stats\_read\_bytes\_total**(count) | Total bytes read for this disk*Shown as byte* | | **portworx.px\_disk\_stats\_read\_seconds\_total**(count) | Total time spent reading in seconds for this disk*Shown as second* | | **portworx.px\_disk\_stats\_write\_seconds\_total**(count) | Total time spent writing in seconds for this disk*Shown as second* | | **portworx.px\_fafb\_apis\_requests\_total**(count) | Total number of API requests made to all the configured FA/FB endpoints | | **portworx.px\_fafb\_apis\_array\_requests\_total**(count) | Total number of API requests made to the /arrays endpoint. | | **portworx.px\_fafb\_apis\_login\_requests\_total**(count) | Total number of API requests made to the /login endpoint. | | **portworx.px\_fafb\_apis\_controllers\_requests\_total**(count) | Total number of API requests made to the /controllers endpoint | | **portworx.px\_fafb\_apis\_alerts\_requests\_total**(count) | Total number of API requests made to the /alerts endpoint | | **portworx.px\_fafb\_apis\_ports\_requests\_total**(count) | Total number of API requests made to the /ports endpoint | | **portworx.px\_fafb\_apis\_volumes\_requests\_total**(count) | Total number of API requests made to the /volumes endpoint | | **portworx.px\_fafb\_apis\_volumesnapshots\_requests\_total**(count) | Total number of API requests made to the /volume-snapshots endpoint | | **portworx.px\_fafb\_apis\_hosts\_requests\_total**(count) | Total number of API requests made to the /hosts endpoint | | **portworx.px\_fafb\_apis\_connections\_requests\_total**(count) | Total number of API requests made to the /connections endpoint | | **portworx.px\_fafb\_apis\_version\_requests\_total**(count) | Total number of API requests made to the /api_version endpoint | | **portworx.px\_kvdb\_get\_requests\_total**(count) | Total number of get requests for a key | | **portworx.px\_kvdb\_snapshot\_requests\_total**(count) | Total number of kvdb snapshot requests for a list of prefixes | | **portworx.px\_kvdb\_put\_requests\_total**(count) | Total number of put requests for a key | | **portworx.px\_kvdb\_create\_requests\_total**(count) | Total number of create requests for a key | | **portworx.px\_kvdb\_update\_requests\_total**(count) | Total number of update requests for a key | | **portworx.px\_kvdb\_enumerate\_requests\_total**(count) | Total number of enumerate requests for a key | | **portworx.px\_kvdb\_delete\_requests\_total**(count) | Total number of delete requests for a key | | **portworx.px\_kvdb\_keys\_requests\_total**(count) | Total number of list key requests for a prefix | | **portworx.px\_kvdb\_cas\_requests\_total**(count) | Total number of compare and sets for a key | | **portworx.px\_kvdb\_cad\_requests\_total**(count) | Total number of compare and deletes for a key | | **portworx.px\_kvdb\_lock\_requests\_total**(count) | Total number of lock requests for a key | | **portworx.px\_kvdb\_unlock\_requests\_total**(count) | Total number of unlock requests for a key | | **portworx.px\_kvdb\_watchkey\_requests\_total**(count) | Total number of watch requests for a key from a node | | **portworx.px\_kvdb\_watchtree\_requests\_total**(count) | Total number of watch requests for a prefix from a node | | **portworx.px\_kvdb\_adduser\_requests\_total**(count) | Total number of add user requests | | **portworx.px\_kvdb\_removeuser\_requests\_total**(count) | Total number of remove user requests | | **portworx.px\_kvdb\_grantuser\_access\_requests\_total**(count) | Total number of grant access requests for a user | | **portworx.px\_kvdb\_revokeuser\_access\_requests\_total**(count) | Total number of revoke access requests for a user | | **portworx.px\_kvdb\_addmember\_requests\_total**(count) | Total number of add member requests from a node | | **portworx.px\_kvdb\_addlearner\_requests\_total**(count) | Total number of add learner requests from a node | | **portworx.px\_kvdb\_removemember\_requests\_total**(count) | Total number of remove member requests from a node | | **portworx.px\_kvdb\_updatemember\_requests\_total**(count) | Total number of update member requests from a node | | **portworx.px\_kvdb\_listmembers\_requests\_total**(count) | Total number of list member requests from a node | | **portworx.px\_kvdb\_setendpoints\_requests\_total**(count) | Total number of set endpoint requests from a node | | **portworx.px\_kvdb\_getendpoints\_requests\_total**(count) | Total number of get endpoint requests from a node | | **portworx.px\_kvdb\_defragment\_requests\_total**(count) | Total number of defragment requests from a node | | **portworx.px\_kvdb\_watch\_latency\_seconds**(gauge) | Time taken in seconds between a kvdb put and a corresponding watch update*Shown as second* | | **portworx.px\_kvdb\_health\_state\_node\_view**(gauge) | This node's kvdb state (1 = healthy, 2 = not healthy) | | **portworx.px\_kvdb\_health\_state\_cluster\_view**(gauge) | This node's view of other node's kvdb state (1 = healthy, 2 = not healthy) | | **portworx.px\_network\_io\_bytessent**(gauge) | Number of bytes sent during this interval*Shown as byte* | | **portworx.px\_network\_io\_received\_bytes**(gauge) | Number of bytes received during this interval*Shown as byte* | | **portworx.px\_network\_io\_sent\_bytes\_total**(count) | Total number of bytes sent*Shown as byte* | | **portworx.px\_network\_io\_received\_bytes\_total**(count) | Total number of bytes received*Shown as byte* | | **portworx.px\_network\_io\_reservation\_waits**(gauge) | Total number of SM waits for reservation | | **portworx.px\_network\_io\_reservation\_wake\_ups**(gauge) | Total number of SM wake ups after getting the reservation | | **portworx.px\_nfs\_pool\_stats\_packets\_arrived**(gauge) | Counts the number of NFS packets arrived on the socket. | | **portworx.px\_nfs\_pool\_stats\_sockets\_enqueued**(gauge) | Counts how many times an NFS transport is enqueued to wait for an nfsd thread to service it, i.e. no nfsd thread was considered available. | | **portworx.px\_nfs\_pool\_stats\_threads\_woken**(gauge) | Counts how many times an idle nfsd thread is woken to try to receive some data from an NFS transport. | | **portworx.px\_nfs\_pool\_stats\_threads\_timedout**(gauge) | Counts how many times an nfsd thread triggered an idle timeout, i.e. was not woken to handle any incoming network packets for some time. | | **portworx.px\_node\_stats\_used\_mem**(gauge) | Used memory in bytes*Shown as byte* | | **portworx.px\_node\_stats\_cpu\_usage**(gauge) | Percent of CPU consumption*Shown as percent* | | **portworx.px\_node\_stats\_busy\_threads\_percent**(gauge) | Percentage of Busy Threads*Shown as percent* | | **portworx.px\_node\_stats\_thread\_pool\_high\_priority\_queue\_size**(gauge) | Thread Pool High Priority Queue Size | | **portworx.px\_node\_stats\_writeback\_list\_num\_dirty\_pools**(gauge) | Writeback List Number of Dirty Pools | | **portworx.px\_node\_stats\_writeback\_list\_num\_pending\_log\_bufs**(gauge) | Writeback List Number of Pending Log Buffers | | **portworx.px\_node\_stats\_writeback\_list\_low\_cos\_queue\_size**(gauge) | Writeback List Low Cos Queue Size | | **portworx.px\_node\_stats\_writeback\_list\_high\_cos\_queue\_size**(gauge) | Writeback List High Cos Queue Size | | **portworx.px\_node\_stats\_sm\_run\_queue\_size**(gauge) | Statemachine Run Queue Size | | **portworx.px\_node\_stats\_kaio\_pending\_queue\_size**(gauge) | Kaio Pending Queue Size | | **portworx.px\_node\_stats\_request\_scheduler\_queue\_size**(gauge) | Request Scheduler Queue Size | | **portworx.px\_node\_stats\_procfs\_mem\_available\_bytes**(gauge) | Available memory in bytes*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_dirty\_bytes**(gauge) | The total amount of memory waiting to be written back to the disk*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_writeback\_bytes**(gauge) | The total amount of memory actively being written back to the disk*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_total\_bytes**(gauge) | Total amount of usable RAM which is physical RAM minus a number of reserved bits and the kernel binary code*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_free\_bytes**(gauge) | The amount of physical RAM, in bytes, left unused by the system.*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_buffers\_bytes**(gauge) | The amount, in bytes, of temporary storage for raw disk blocks*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_cached\_bytes**(gauge) | The amount of physical RAM, in bytes, used as cache memory*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_active\_anon\_bytes**(gauge) | The amount of anonymous and tmpfs/shmem memory, in bytes, that is in active use*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_inactive\_anon\_bytes**(gauge) | The amount of anonymous and tmpfs/shmem memory, in bytes, that is a candidate for eviction*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_active\_file\_bytes**(gauge) | The amount of file cache memory, in bytes, that is in active use*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_inactive\_file\_bytes**(gauge) | The amount of file cache memory, in bytes, that is newly loaded from the disk*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_unevictable\_bytes**(gauge) | The amount of memory, in bytes, discovered by the pageout code*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_mlocked\_bytes**(gauge) | The total amount of memory, in bytes, that is not evictable because it is locked into memory by user programs*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_anon\_pages\_bytes**(gauge) | The total amount of memory, in bytes, used by pages that are not backed by files and are mapped into userspace page tables*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_mapped\_bytes**(gauge) | The memory, in bytes, used for files that have been mmaped*Shown as byte* | | **portworx.px\_node\_stats\_procfs\_mem\_sh\_mem\_bytes**(gauge) | The total amount of memory, in bytes, used by shared memory (shmem) and tmpfs*Shown as byte* | | **portworx.px\_node\_stats\_num\_skinnysnaps**(gauge) | The number of SkinnySnaps | | **portworx.px\_node\_stats\_skinnysnaps\_num\_repls\_skips**(gauge) | The number of target replicas skipped for a snapshot due to SkinnySnap | | **portworx.px\_node\_stats\_skinnysnaps\_num\_repls\_snapshots**(gauge) | The number of target replicas that underwent a snapshot operation due to SkinnySnap | | **portworx.px\_node\_stats\_relaxed\_reclaim\_pending**(gauge) | The number of volumes in RelaxedReclaim pending queue | | **portworx.px\_node\_stats\_relaxed\_reclaim\_skipped**(gauge) | The number of volume deletes that were skipped from being delayed even when RelaxedReclaim was enabled. | | **portworx.px\_node\_stats\_relaxed\_reclaim\_deleted**(gauge) | The number of volume deletes done through RelaxedReclaim | | **portworx.px\_node\_stats\_num\_volumes**(gauge) | The number of volumes hosted on this node | | **portworx.px\_node\_status\_node\_status**(gauge) | Status of this node ([https://libopenstorage.github.io/w/master.generated-api/#status](https://libopenstorage.github.io/w/master.generated-api/#status)) | | **portworx.px\_node\_status\_license\_expiry**(gauge) | Number of days until License (or License lease) expires (<0 means Expired) | | **portworx.px\_none\_stats\_free\_mem**(gauge) | Available memory in bytes*Shown as byte* | | **portworx.px\_none\_stats\_total\_mem**(gauge) | Total memory in bytes*Shown as byte* | | **portworx.px\_pool\_stats\_pool\_written\_bytes**(gauge) | Bytes written since last interval for this pool. **Deprecated**.*Shown as byte* | | **portworx.px\_pool\_stats\_pool\_write\_latency\_seconds**(gauge) | Average time spent per write operation for this pool. **Deprecated**.*Shown as second* | | **portworx.px\_pool\_stats\_pool\_writethroughput**(gauge) | Average number of bytes written per second for this pool. **Deprecated**.*Shown as byte* | | **portworx.px\_pool\_stats\_pool\_flushed\_bytes**(gauge) | Number of flushed bytes since last interval for this pool. **Deprecated**.*Shown as byte* | | **portworx.px\_pool\_stats\_pool\_num\_flushes**(gauge) | Number of flush(sync) operations since last interval for this pool. **Deprecated**. | | **portworx.px\_pool\_stats\_pool\_flushms**(gauge) | Latency for flush for this pool. **Deprecated**.*Shown as millisecond* | | **portworx.px\_pool\_stats\_pool\_provisioned\_bytes**(gauge) | Provisioned storage space in bytes for this pool. **Deprecated**.*Shown as byte* | | **portworx.px\_pool\_stats\_pool\_status**(gauge) | Status of this Pool (0=Offline,1=Online). **Deprecated**. | | **portworx.px\_pool\_stats\_written\_bytes**(gauge) | Bytes written since last interval for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_num\_writes**(gauge) | Number of write operations in the last interval for this pool | | **portworx.px\_pool\_stats\_write\_ms**(gauge) | Total time in millisecond spent in writing in the last interval for this pool*Shown as millisecond* | | **portworx.px\_pool\_stats\_write\_latency\_seconds**(gauge) | Average time spent per write operation for this pool*Shown as second* | | **portworx.px\_pool\_stats\_write\_iops**(gauge) | Average number of completed write operations per second for this pool | | **portworx.px\_pool\_stats\_writethroughput**(gauge) | Average number of bytes written per second for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_flushed\_bytes**(gauge) | Number of flushed bytes since last interval for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_num\_flushes**(gauge) | Number of flush(sync) operations since last interval for this pool | | **portworx.px\_pool\_stats\_flushms**(gauge) | Latency for flush for this pool*Shown as millisecond* | | **portworx.px\_pool\_stats\_provisioned\_bytes**(gauge) | Provisioned storage space in bytes for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_status**(gauge) | Status of this Pool (0=Offline, 1=Online, 2=Full, 3=NotFound, 4=Maintenance) | | **portworx.px\_pool\_stats\_available\_bytes**(gauge) | Available storage space in bytes for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_used\_bytes**(gauge) | Used storage space in bytes for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_total\_bytes**(gauge) | Total storage space in bytes for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_written\_bytes\_total**(count) | Total bytes written for this pool*Shown as byte* | | **portworx.px\_pool\_stats\_flushed\_bytes\_total**(count) | Total number of flushed bytes*Shown as byte* | | **portworx.px\_pool\_stats\_num\_flushes\_total**(count) | Total number of flush(sync) operations | | **portworx.px\_pool\_stats\_flushms\_total**(count) | Total time spent in flush*Shown as millisecond* | | **portworx.px\_proc\_stats\_virt**(gauge) | Virtual memory in bytes*Shown as byte* | | **portworx.px\_proc\_stats\_res**(gauge) | Resident set size memory in bytes*Shown as byte* | | **portworx.px\_proc\_stats\_cputime**(gauge) | Amount of time that this process has been scheduled in user and kernel mode measured in clock ticks | | **portworx.px\_px\_cache\_status**(gauge) | Cache enabled (0=No,1=Yes) | | **portworx.px\_px\_cache\_total\_blocks**(gauge) | Number of total blocks in the cache | | **portworx.px\_px\_cache\_used\_blocks**(gauge) | Number of used blocks in the cache | | **portworx.px\_px\_cache\_dirty\_blocks**(gauge) | Number of dirty blocks in the cache | | **portworx.px\_px\_cache\_read\_hits**(gauge) | Number of read hits for the cache | | **portworx.px\_px\_cache\_read\_miss**(gauge) | Number of read misses for the cache | | **portworx.px\_px\_cache\_write\_hits**(gauge) | Number of write hits for the cache | | **portworx.px\_px\_cache\_write\_miss**(gauge) | Number of write misses for the cache | | **portworx.px\_px\_cache\_block\_size**(gauge) | Block size for the cache | | **portworx.px\_px\_cache\_mode**(gauge) | Mode of the cache | | **portworx.px\_px\_cache\_migrate\_promote**(gauge) | Number of blocks promoted to the cache | | **portworx.px\_px\_cache\_migrate\_demote**(gauge) | Number of block demoted from the cache | | **portworx.px\_px\_cache\_io\_mbps**(gauge) | Approximate cache bandwidth from cache internal actions | | **portworx.px\_rebalance\_rebalance\_job\_state**(gauge) | Rebalance job state (0 = pending, 1 = running, 2 = done, 3 = paused, 4 = cancelled) | | **portworx.px\_rebalance\_provision\_space\_rebalanced\_bytes\_total**(count) | Total provisioned space rebalanced (only counts add (since remove has equal value as add))*Shown as byte* | | **portworx.px\_rebalance\_used\_space\_rebalanced\_bytes\_total**(count) | Total used space rebalanced (only counts add (since remove has equal value as add))*Shown as byte* | | **portworx.px\_rebalance\_volumes\_rebalanced\_total**(count) | Total volumes affected by rebalance operation | | **portworx.px\_rebalance\_overloaded\_pools\_total**(count) | Number of overloaded pools | | **portworx.px\_volume\_usage\_bytes**(gauge) | Used storage space in bytes for this volume*Shown as byte* | | **portworx.px\_volume\_capacity\_bytes**(gauge) | Configured size in bytes for this volume*Shown as byte* | | **portworx.px\_volume\_halevel**(gauge) | Configured HA level for this volume | | **portworx.px\_volume\_currhalevel**(gauge) | Current HA level for this volume | | **portworx.px\_volume\_iopriority**(gauge) | Configured I/O priority for this volume | | **portworx.px\_volume\_elapsed\_time\_since\_detached\_seconds**(gauge) | Seconds elapsed since the volume is detached*Shown as second* | | **portworx.px\_volume\_elapsed\_time\_since\_attached\_seconds**(gauge) | Seconds elapsed since the volume is attached*Shown as second* | | **portworx.px\_volume\_attached**(gauge) | Attached state for this volume (0=detached,1=attached) | | **portworx.px\_volume\_status**(gauge) | Status for this volume ([https://libopenstorage.github.io/w/master.generated-api/#volumestatus](https://libopenstorage.github.io/w/master.generated-api/#volumestatus)) | | **portworx.px\_volume\_state**(gauge) | State for this volume ([https://libopenstorage.github.io/w/master.generated-api/#volumestate](https://libopenstorage.github.io/w/master.generated-api/#volumestate)) | | **portworx.px\_volume\_attached\_state**(gauge) | Attached state for this volume (valid only if volume is attached) ([https://libopenstorage.github.io/w/master.generated-api/#attachstate](https://libopenstorage.github.io/w/master.generated-api/#attachstate)) | | **portworx.px\_volume\_fs\_health\_status**(gauge) | Filesystem health status for this volume ([https://libopenstorage.github.io/w/master.generated-api/#filesystemhealthstatus](https://libopenstorage.github.io/w/master.generated-api/#filesystemhealthstatus)) | | **portworx.px\_volume\_replication\_status**(gauge) | Replication Status for this volume (0 : up, 1 : not in quorum, 2 : resync state, 3 : degraded, 4 : detached, 5 : restore) | | **portworx.px\_volume\_fs\_usage\_bytes**(gauge) | Used storage space in bytes as reported by the filesystem for this volume*Shown as byte* | | **portworx.px\_volume\_fs\_capacity\_bytes**(gauge) | Total storage space in bytes as reported by the filesystem for this volume*Shown as byte* | | **portworx.px\_volume\_vol\_read\_bytes**(gauge) | Number of successfully read bytes during this interval for this volume. **Deprecated**.*Shown as byte* | | **portworx.px\_volume\_vol\_written\_bytes**(gauge) | Number of successfully written bytes during this interval for this volume. **Deprecated**.*Shown as byte* | | **portworx.px\_volume\_vol\_reads**(gauge) | Number of successfully completed read operations during this interval for this volume. **Deprecated**. | | **portworx.px\_volume\_vol\_writes**(gauge) | Number of successfully completed write operations during this interval for this volume. **Deprecated**. | | **portworx.px\_volume\_read\_bytes**(gauge) | Number of successfully read bytes during this interval for this volume*Shown as byte* | | **portworx.px\_volume\_written\_bytes**(gauge) | Number of successfully written bytes during this interval for this volume*Shown as byte* | | **portworx.px\_volume\_reads**(gauge) | Number of successfully completed read operations during this interval for this volume | | **portworx.px\_volume\_writes**(gauge) | Number of successfully completed write operations during this interval for this volume | | **portworx.px\_volume\_read\_bytes\_total**(count) | Total number of successfully read bytes for this volume*Shown as byte* | | **portworx.px\_volume\_written\_bytes\_total**(count) | Total number of successfully written bytes for this volume*Shown as byte* | | **portworx.px\_volume\_reads\_total**(count) | Total number of successfully completed read operations for this volume | | **portworx.px\_volume\_writes\_total**(count) | Total number of successfully completed write operations for this volume | | **portworx.px\_volume\_iops**(gauge) | Number of successful completed I/O operations per second during this interval for this volume | | **portworx.px\_volume\_write\_iops**(gauge) | Average number of completed write operations per second for this volume | | **portworx.px\_volume\_read\_iops**(gauge) | Average number of completed read operations per second for this volume | | **portworx.px\_volume\_vol\_num\_sequential\_writes**(gauge) | Number of sequential write I/O operations during this interval for this volume | | **portworx.px\_volume\_vol\_num\_sequential\_reads**(gauge) | Number of sequential read I/O operations during this interval for this volume | | **portworx.px\_volume\_vol\_num\_random\_writes**(gauge) | Number of random write I/O operations during this interval for this volume | | **portworx.px\_volume\_vol\_num\_random\_reads**(gauge) | Number of random read I/O operations during this interval for this volume | | **portworx.px\_volume\_depth\_io**(gauge) | Number of I/O operations currently in progress for this volume | | **portworx.px\_volume\_readthroughput**(gauge) | Number of bytes read per second during this interval for this volume*Shown as byte* | | **portworx.px\_volume\_writethroughput**(gauge) | Number of bytes written per second during this interval for this volume*Shown as byte* | | **portworx.px\_volume\_vol\_bytes\_reclaimed**(gauge) | Number of bytes reclaimed by fstrim operation*Shown as byte* | | **portworx.px\_volume\_vol\_bytes\_reclaimable**(gauge) | Number of bytes reclaimable on the volume as seen by fstrim operation*Shown as byte* | | **portworx.px\_volume\_vol\_read\_latency\_seconds**(gauge) | Average time spent per successfully completed read operation in seconds during this interval for this volume. **Deprecated**.*Shown as second* | | **portworx.px\_volume\_vol\_write\_latency\_seconds**(gauge) | Average time spent per successfully completed write operation in seconds during this interval for this volume. **Deprecated**.*Shown as second* | | **portworx.px\_volume\_read\_latency\_seconds**(gauge) | Average time spent per successfully completed read operation in seconds for this volume*Shown as second* | | **portworx.px\_volume\_write\_latency\_seconds**(gauge) | Average time spent per successfully completed write operation in seconds for this volume*Shown as second* | | **portworx.px\_volume\_num\_long\_reads**(gauge) | Number of long reads for this volume | | **portworx.px\_volume\_num\_long\_writes**(gauge) | Number of long writes for this volume | | **portworx.px\_volume\_num\_long\_flushes**(gauge) | Number of long flushes for this volume | | **portworx.px\_volume\_num\_db\_flushes**(gauge) | Number of DB flushes for this volume | | **portworx.px\_volume\_replica\_read\_bytes\_total**(count) | Total number of successfully read bytes for this replica volume*Shown as byte* | | **portworx.px\_volume\_replica\_written\_bytes\_total**(count) | Total number of successfully written bytes for this replica volume*Shown as byte* | | **portworx.px\_volume\_replica\_reads\_total**(count) | Total number of successfully completed read operations for this replica volume | | **portworx.px\_volume\_replica\_writes\_total**(count) | Total number of successfully completed write operations for this replica volume | | **portworx.px\_volume\_replica\_flushes\_total**(count) | Total number of successfully completed flush operations for this replica volume | | **portworx.px\_volume\_replica\_read\_ms\_total**(count) | Total time spent doing read operations in milliseconds for this replica volume*Shown as millisecond* | | **portworx.px\_volume\_replica\_write\_ms\_total**(count) | Total time doing write operations in milliseconds for this replica volume*Shown as millisecond* | | **portworx.px\_volume\_replica\_flush\_ms\_total**(count) | Total time doing write operations in milliseconds for this replica volume*Shown as millisecond* | | **portworx.px\_volume\_dev\_depth\_io**(gauge) | Number of I/O operations currently in progress as reported by the kernel pxd device for this volume | | **portworx.px\_volume\_dev\_writethroughput**(gauge) | Number of successfully written bytes per second as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px\_volume\_dev\_readthroughput**(gauge) | Number of successfully read bytes per second as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px\_volume\_dev\_read\_latency\_seconds**(gauge) | Average time spent per successfully completed read in seconds as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px\_volume\_dev\_write\_latency\_seconds**(gauge) | Average time spent per successfully completed write in seconds as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px\_volume\_dev\_read\_bytes\_total**(count) | Total number of successfully read bytes as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px\_volume\_dev\_written\_bytes\_total**(count) | Number of successfully written bytes as reported by the kernel pxd device for this volume*Shown as byte* | | **portworx.px\_volume\_dev\_reads\_total**(count) | Total number of successfully completed read operations as reported by the kernel pxd device for this volume | | **portworx.px\_volume\_dev\_writes\_total**(count) | Total number of successfully completed write operations as reported by the kernel pxd device for this volume | | **portworx.px\_volume\_dev\_read\_seconds\_total**(count) | Total time spent reading in seconds for this disk as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px\_volume\_dev\_write\_seconds\_total**(count) | Total time spent writing in seconds for this disk as reported by the kernel pxd device for this volume*Shown as second* | | **portworx.px\_volume\_unique\_blocks**(gauge) | Size(in bytes) of unique blocks for this volume*Shown as byte* | | **portworx.px\_volume\_timestamp\_records**(gauge) | Number of timestamp records accumulated | | **portworx.px\_volume\_timestamp\_records\_per\_node**(gauge) | Number of timestamp records accumulated for a node | | **portworx.px\_volume\_num\_discard\_ops**(gauge) | Number of discard operations for this volume | | **portworx.px\_volume\_discarded\_bytes**(gauge) | Number of discarded bytes for this volume*Shown as byte* | | **portworx.autopilot\_errors\_total**(count) | Total number of autopilot errors | | **portworx.autopilot\_action\_failed\_count**(gauge) | Total number of declined or failed actions | | **portworx.autopilot\_action\_failed\_total**(count) | Total number of failed actions per rule object | | **portworx.autopilot\_objects\_in\_stable\_condition\_total**(count) | Total number of objects with stable conditions per rule | | **portworx.autopilot\_action\_taken\_total**(count) | Total number of actions taken per rule object | | **portworx.autopilot\_objects\_total**(count) | Total number of objects per rule | | **portworx.autopilot\_objects\_with\_rules\_total**(count) | Total number of objects matching any rules | | **portworx.autopilot\_rules\_in\_system**(gauge) | Total number of rules in the system | | **portworx.autopilot\_dynamic\_throttle\_applied\_total**(count) | Total number of dynamic throttles applied | | **portworx.autopilot\_dynamic\_throttle\_failed\_total**(count) | Total number of dynamic throttles failed to apply | | **portworx.autopilot\_dynamic\_throttle\_skipped\_total**(count) | Total number of dynamic throttles skipped to apply | | **portworx.autopilot\_datadog\_success\_total**(count) | Total number of successful Datadog queries | | **portworx.autopilot\_datadog\_failures\_total**(count) | Total number of failed Datadog queries | | **portworx.autopilot\_datadog\_query\_validation\_failures\_total**(count) | Total number of failed Datadog query validations | | **portworx.autopilot\_datadog\_empty\_response\_total**(count) | Total number of empty Datadog responses | | **portworx.autopilot\_prometheus\_empty\_response\_total**(count) | Total number of empty Prometheus responses | | **portworx.stork\_migration\_status**(gauge) | Migration status - used to filter and list migrations by status within the cluster. | | **portworx.stork\_migration\_stage**(gauge) | Stage of migration (0 = Initial, 1 = PreExecRule, 2 = PostExecRule, 3 = Volumes, 4 = Applications, 5 = Final). | | **portworx.stork\_migration\_duration**(gauge) | Duration of migration in seconds. Stage label values: `volume` (volume migration) or `resource` (resource migration).*Shown as second* | | **portworx.stork\_migration\_duration\_seconds**(gauge) | Total time taken by a migration in seconds.*Shown as second* | | **portworx.stork\_migration\_resources**(gauge) | Count of resources collected for migration (state: planned or migrated). | | **portworx.stork\_migration\_volumes**(gauge) | Volumes planned for migration (state: planned or migrated). | | **portworx.stork\_migration\_created\_total**(count) | Increments when a migration is created. | | **portworx.stork\_migration\_completed\_total**(count) | Increments when a migration reaches a terminal state (Successful, Failed, or PartialSuccess). | | **portworx.stork\_migration\_transfer\_rate**(gauge) | Data transfer rate for a single migration (bytes/second).*Shown as byte* | | **portworx.stork\_migration\_transferred\_bytes**(gauge) | Total bytes transferred during a single migration.*Shown as byte* | | **portworx.stork\_migration\_completion\_timestamp\_seconds**(gauge) | Timestamp when the latest migration completed for a schedule with the status being Successful, PartialSuccess, or Failed.*Shown as second* | | **portworx.stork\_migration\_schedule\_status**(gauge) | Status of migration schedules. | | **portworx.stork\_migration\_schedule\_last\_run\_status**(gauge) | Status of the latest run of each migration schedule. | | **portworx.stork\_action\_status**(gauge) | Status of Failover or Failback actions. | | **portworx.stork\_action\_created\_total**(count) | Increments when a Failover/Failback action is created. | | **portworx.stork\_action\_completed\_total**(count) | Increments when a Failover/Failback action reaches a terminal state. | | **portworx.stork\_action\_completion\_timestamp\_seconds**(gauge) | Timestamp when a Failover/Failback action completes with the status as Successful, Failed, RollbackSuccessful, or RollbackFailed.*Shown as second* | | **portworx.stork\_clusterpair\_scheduler\_status**(gauge) | Status of the cluster pair's health with respect to Kubernetes (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). | | **portworx.stork\_clusterpair\_storage\_status**(gauge) | Status of the cluster pair's health with respect to storage provisioner (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). | | **portworx.stork\_all\_namespaces**(gauge) | Number of namespaces in the Kubernetes cluster, including Kubernetes system namespaces. | | **portworx.stork\_protected\_namespaces**(gauge) | Count of namespaces being migrated through active migration schedules. | | **portworx.stork\_hyperconverged\_pods\_total**(gauge) | Number of pods scheduled on a node where replicas for all pod volumes exist (hyper-converged by the Stork scheduler). | | **portworx.stork\_non\_hyperconverged\_pods\_total**(gauge) | Number of pods not hyper-converged by the Stork scheduler. | | **portworx.stork\_semi\_hyperconverged\_pods\_total**(gauge) | Number of pods partially hyper-converged (scheduled on nodes with some but not all volume replicas). | ### Events{% #events %} The Portworx integration does not include any events. ### Service Checks{% #service-checks %} The Portworx integration does not include any service checks. ## Troubleshooting{% #troubleshooting %} ### Agent cannot scrape metrics{% #agent-cannot-scrape-metrics %} If metrics are not appearing in Datadog, check the following: - Verify the `ComponentK8sConfig` CR was applied and the pods have the expected annotations: ```shell kubectl get pod -n portworx -o jsonpath='{.metadata.annotations}' ``` - Confirm the OpenMetrics endpoints are reachable from within the Agent pod: ```shell kubectl exec -n -- curl http://:17001/metrics ``` - Check Agent logs for scrape errors: ```shell kubectl logs -n ``` ## Support{% #support %} Need help? Contact [Portworx support](mailto:support@purestorage.com). ## Further Reading{% #further-reading %} - [Portworx + Datadog: Monitor your complete container infrastructure](https://www.datadoghq.com/blog/portworx-integration/)