--- title: StreamNative description: Gain insights into StreamNative metrics data. breadcrumbs: Docs > Integrations > StreamNative --- # StreamNative Integration version1.0.0 StreamNative - HealthStreamNative - Kafka ConnectStreamNative - Pulsar ResourceStreamNative - Sink ConnectorStreamNative - Source Connector ## Overview{% #overview %} [StreamNative](https://streamnative.io/) provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services. The StreamNative integration collects the following types of [metrics](https://docs.streamnative.io/docs/cloud-metrics-api#metrics-endpoint): 1. Health 1. Pulsar Resource 1. Source Connector 1. Sink Connector 1. Kafka Connect ## Setup{% #setup %} ### Generate API credentials in StreamNative{% #generate-api-credentials-in-streamnative %} 1. Log into [StreamNative Cloud Console Account](https://console.streamnative.cloud/). 1. Click the profile icon and navigate to the **Accounts & Accesses** tab. 1. Find the Service Account with **Admin** permissions or a binding to the `metrics-viewer` role. - If no Service Account exists, select **New -> Service Account** to create one, and make sure to enable the **Super Admin** option. - To bind a service account with the `metrics-viewer` role, refer to the [metrics-viewer rolebinding](https://docs.streamnative.io/docs/cloud-metrics-api#metrics-authorization) documentation. 1. On the right side of the chosen Service Account, click the `...` button. 1. Select **Download OAuth2 Key** to obtain the **Client ID** and **Client Secret**. ### Get `Organization ID` and `Instance Name`{% #get-organization-id-and-instance-name %} 1. Click the profile icon and select **Organizations**. 1. Choose the **Organization** for which data needs to be collected. 1. From the **Select an Instance** dropdown, get the **Instance Name**. ### Connect your StreamNative account to Datadog{% #connect-your-streamnative-account-to-datadog %} 1. Add your organization ID, instance name, client ID, and client secret. | Parameters | Description | | --------------- | ------------------------------------------------ | | Organization ID | Organization ID of your StreamNative account. | | Instance Name | Instance name of the corresponding organization. | | Client ID | Client ID of your service account. | | Client Secret | Client secret of your service account. | 1. Click the **Save** button to save your settings. ## Data Collected{% #data-collected %} ### Logs{% #logs %} The StreamNative integration does not include any logs. ### Metrics{% #metrics %} | | | | | **streamnative.health.pulsar\_detector\_e2e\_latency\_ms**(gauge) | The latency distribution from message sending to message consumption*Shown as millisecond* | | **streamnative.health.pulsar\_detector\_geo\_latency\_ms**(gauge) | The latency distribution Latency distribution from message sending to message consumption across clusters*Shown as millisecond* | | **streamnative.health.pulsar\_detector\_publish\_latency\_ms**(gauge) | The latency distribution of message sending*Shown as millisecond* | | **streamnative.health.pulsar\_detector\_pulsar\_sla\_messaging\_up**(gauge) | The gauge for indicating the messaging service up or down | | **streamnative.health.pulsar\_detector\_pulsar\_sla\_webservice\_up**(gauge) | The gauge for indicating the webservice up or down | | **streamnative.kafka\_connect.jvm\_gc\_collection\_seconds\_sum**(gauge) | Time spent in a given JVM garbage collector in seconds.*Shown as second* | | **streamnative.kafka\_connect.jvm\_memory\_committed\_bytes**(gauge) | Committed bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka\_connect.jvm\_memory\_init\_bytes**(gauge) | Initial bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka\_connect.jvm\_memory\_max\_bytes**(gauge) | Max bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka\_connect.jvm\_memory\_used\_bytes**(gauge) | Used bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_batch\_size\_avg**(gauge) | The average size of the batches processed by the connector | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_batch\_size\_max**(gauge) | The maximum size of the batches processed by the connector | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_offset\_commit\_avg\_time\_ms**(gauge) | The average time in milliseconds taken by this task to commit offsets*Shown as millisecond* | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_offset\_commit\_failure\_percentage**(gauge) | The average percentage of this task's offset commit attempts that failed*Shown as fraction* | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_offset\_commit\_max\_time\_ms**(gauge) | The maximum time in milliseconds taken by this task to commit offsets*Shown as millisecond* | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_offset\_commit\_success\_percentage**(gauge) | The average percentage of this task's offset commit attempts that succeeded*Shown as fraction* | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_pause\_ratio**(gauge) | The fraction of time this task has spent in the pause state*Shown as fraction* | | **streamnative.kafka\_connect.kafka\_connect\_connector\_task\_running\_ratio**(gauge) | The fraction of time this task has spent in the running state*Shown as fraction* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_offset\_commit\_completion**(gauge) | The total number of offset commit completions that were completed successfully*Shown as offset* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_offset\_commit\_completion\_rate**(gauge) | The average per-second number of offset commit completions that were completed successfully*Shown as offset* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_offset\_commit\_seq\_no**(gauge) | The current sequence number for offset commits | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_offset\_commit\_skip**(gauge) | The total number of offset commit completions that were received too late and skipped/ignored*Shown as offset* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_offset\_commit\_skip\_rate**(gauge) | The average per-second number of offset commit completions that were received too late and skipped/ignored*Shown as offset* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_partition\_count**(gauge) | The number of topic partitions assigned to this task belonging to the named sink connector in this worker | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_put\_batch\_avg\_time\_ms**(gauge) | The average time taken by this task to put a batch of sinks records*Shown as millisecond* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_put\_batch\_max\_time\_ms**(gauge) | The maximum time taken by this task to put a batch of sinks records*Shown as millisecond* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_sink\_record\_active\_count**(gauge) | The number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_sink\_record\_active\_count\_avg**(gauge) | The average number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_sink\_record\_active\_count\_max**(gauge) | The maximum number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_sink\_record\_read**(gauge) | The total number of records read from Kafka by this task belonging to the named sink connector in this worker, since the task was last restarted*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_sink\_record\_read\_rate**(gauge) | The average per-second number of records read from Kafka for this task belonging to the named sink connector in this worker. This is before transformations are applied*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_sink\_record\_send**(gauge) | The total number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker, since the task was last restarted*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_sink\_task\_sink\_record\_send\_rate**(gauge) | The average per-second number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_poll\_batch\_avg\_time\_ms**(gauge) | The average time in milliseconds taken by this task to poll for a batch of source records*Shown as millisecond* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_poll\_batch\_max\_time\_ms**(gauge) | The maximum time in milliseconds taken by this task to poll for a batch of source records*Shown as millisecond* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_source\_record\_active\_count**(gauge) | The number of records that have been produced by this task but not yet completely written to Kafka*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_source\_record\_active\_count\_avg**(gauge) | The average number of records that have been produced by this task but not yet completely written to Kafka*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_source\_record\_active\_count\_max**(gauge) | The maximum number of records that have been produced by this task but not yet completely written to Kafka*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_source\_record\_poll**(gauge) | The total number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_source\_record\_poll\_rate**(gauge) | The average per-second number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_source\_record\_write**(gauge) | The number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker, since the task was last restarted*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_source\_task\_source\_record\_write\_rate**(gauge) | The average per-second number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_deadletterqueue\_produce\_failures**(gauge) | The number of failed writes to the dead letter queue | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_deadletterqueue\_produce\_requests**(gauge) | The number of attempted writes to the dead letter queue | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_last\_error\_timestamp**(gauge) | The epoch timestamp when this task last encountered an error*Shown as millisecond* | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_total\_errors\_logged**(gauge) | The total number of errors that were logged*Shown as error* | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_total\_record\_errors**(gauge) | The total number of record processing errors in this task*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_total\_record\_failures**(gauge) | The total number of record processing failures in this task*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_total\_records\_skipped**(gauge) | The total number of records skipped due to errors*Shown as record* | | **streamnative.kafka\_connect.kafka\_connect\_task\_error\_total\_retries**(gauge) | The total number of operations retried*Shown as operation* | | **streamnative.kafka\_connect.kafka\_connect\_worker\_connector\_destroyed\_task\_count**(gauge) | The number of destroyed tasks of the connector on the worker*Shown as task* | | **streamnative.kafka\_connect.kafka\_connect\_worker\_connector\_failed\_task\_count**(gauge) | The number of failed tasks of the connector on the worker*Shown as task* | | **streamnative.kafka\_connect.kafka\_connect\_worker\_connector\_paused\_task\_count**(gauge) | The number of paused tasks of the connector on the worker*Shown as task* | | **streamnative.kafka\_connect.kafka\_connect\_worker\_connector\_restarting\_task\_count**(gauge) | The number of restarting tasks of the connector on the worker*Shown as task* | | **streamnative.kafka\_connect.kafka\_connect\_worker\_connector\_running\_task\_count**(gauge) | The number of running tasks of the connector on the worker*Shown as task* | | **streamnative.kafka\_connect.kafka\_connect\_worker\_connector\_total\_task\_count**(gauge) | The number of tasks of the connector on the worker*Shown as task* | | **streamnative.kafka\_connect.kafka\_connect\_worker\_connector\_unassigned\_task\_count**(gauge) | The number of unassigned tasks of the connector on the worker*Shown as task* | | **streamnative.kafka\_connect.process\_cpu\_seconds\_total**(gauge) | Total user and system CPU time spent in seconds*Shown as second* | | **streamnative.pulsar\_resource.pulsar\_consumers\_count**(gauge) | The number of active consumers of the topic connected to this broker. | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_100\_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 100 kilobytes*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_128**(gauge) | The entry rate for a topic with entry size smaller than or equal to 128 bytes*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_16\_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 16 kilobytes*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_1\_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 1 kilobyte*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_1\_mb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 1 megabyte*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_2\_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 2 kilobytes*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_4\_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 4 kilobytes*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_512**(gauge) | The entry rate for a topic with entry size smaller than or equal to 512 bytes*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_entry\_size\_le\_overflow**(gauge) | The entry rate for a topic with entry size greater than 1 megabyte*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_producers\_count**(gauge) | The number of active producers of the topic connected to this broker | | **streamnative.pulsar\_resource.pulsar\_rate\_in**(gauge) | The total message rate of the namespace coming into this broker*Shown as message* | | **streamnative.pulsar\_resource.pulsar\_rate\_out**(gauge) | The total message rate of the namespace going out from this broker*Shown as message* | | **streamnative.pulsar\_resource.pulsar\_storage\_backlog\_size**(gauge) | The total backlog size of the topics of this topic owned by this broker*Shown as byte* | | **streamnative.pulsar\_resource.pulsar\_storage\_offloaded\_size**(gauge) | The total amount of the data in this topic offloaded to the tiered storage*Shown as byte* | | **streamnative.pulsar\_resource.pulsar\_storage\_read\_rate**(gauge) | The total message batches (entries) read from the storage for this topic | | **streamnative.pulsar\_resource.pulsar\_storage\_size**(gauge) | The total storage size of the topics in this topic owned by this broker*Shown as byte* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_0\_5**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 0.5 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_1**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 1 millisecond*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_10**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 10 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_100**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 100 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_1000**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 1000 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_20**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 20 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_200**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 200 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_5**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 5 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_50**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 50 milliseconds*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_latency\_le\_overflow**(gauge) | The entry rate for a topic where the storage write latency is greater than 1 second*Shown as fraction* | | **streamnative.pulsar\_resource.pulsar\_storage\_write\_rate**(gauge) | The total message batches (entries) written to the storage for this topic | | **streamnative.pulsar\_resource.pulsar\_subscription\_delayed**(gauge) | The total message batches (entries) are delayed for dispatching | | **streamnative.pulsar\_resource.pulsar\_subscriptions\_count**(gauge) | The number of Pulsar subscriptions of the topic served by this broker | | **streamnative.pulsar\_resource.pulsar\_throughput\_in**(gauge) | The total throughput of the topic coming into this broker*Shown as byte* | | **streamnative.pulsar\_resource.pulsar\_throughput\_out**(gauge) | The total throughput of the topic going out from this broker*Shown as byte* | | **streamnative.pulsar\_resource.pulsar\_topics\_count**(gauge) | The number of Pulsar topics of the namespace owned by this broker | | **streamnative.sink\_connector.jvm\_gc\_collection\_seconds\_sum**(gauge) | Time spent in a given JVM garbage collector in seconds*Shown as second* | | **streamnative.sink\_connector.jvm\_memory\_bytes\_committed**(gauge) | Committed bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink\_connector.jvm\_memory\_bytes\_init**(gauge) | Initial bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink\_connector.jvm\_memory\_bytes\_max**(gauge) | Max bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink\_connector.jvm\_memory\_direct\_bytes\_used**(gauge) | Used bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink\_connector.process\_cpu\_seconds\_total**(gauge) | Total user and system CPU time spent in seconds*Shown as second* | | **streamnative.sink\_connector.pulsar\_sink\_last\_invocation**(gauge) | The timestamp of the last invocation of the sink*Shown as millisecond* | | **streamnative.sink\_connector.pulsar\_sink\_received\_1min\_total**(gauge) | The total number of records received from sink in the last 1 minute*Shown as record* | | **streamnative.sink\_connector.pulsar\_sink\_received\_total**(gauge) | The total number of records received from sink*Shown as record* | | **streamnative.sink\_connector.pulsar\_sink\_sink\_exception**(gauge) | The exception from a sink | | **streamnative.sink\_connector.pulsar\_sink\_sink\_exceptions\_1min\_total**(gauge) | The total number of sink exceptions in the last 1 minute*Shown as exception* | | **streamnative.sink\_connector.pulsar\_sink\_sink\_exceptions\_total**(gauge) | The total number of sink exceptions*Shown as exception* | | **streamnative.sink\_connector.pulsar\_sink\_system\_exception**(gauge) | The exception from system code | | **streamnative.sink\_connector.pulsar\_sink\_system\_exceptions\_1min\_total**(gauge) | The total number of system exceptions in the last 1 minute*Shown as exception* | | **streamnative.sink\_connector.pulsar\_sink\_system\_exceptions\_total**(gauge) | The total number of system exceptions*Shown as exception* | | **streamnative.sink\_connector.pulsar\_sink\_written\_1min\_total**(gauge) | The total number of records written to a Pulsar topic in the last 1 minute*Shown as record* | | **streamnative.sink\_connector.pulsar\_sink\_written\_total**(gauge) | The total number of records written to a Pulsar topic*Shown as record* | | **streamnative.source\_connector.jvm\_gc\_collection\_seconds\_sum**(gauge) | Time spent in a given JVM garbage collector in seconds*Shown as second* | | **streamnative.source\_connector.jvm\_memory\_bytes\_committed**(gauge) | Committed bytes of a given JVM memory area*Shown as byte* | | **streamnative.source\_connector.jvm\_memory\_bytes\_init**(gauge) | Initial bytes of a given JVM memory area*Shown as byte* | | **streamnative.source\_connector.jvm\_memory\_bytes\_max**(gauge) | Max bytes of a given JVM memory area*Shown as byte* | | **streamnative.source\_connector.jvm\_memory\_direct\_bytes\_used**(gauge) | Used bytes of a given JVM memory area*Shown as byte* | | **streamnative.source\_connector.process\_cpu\_seconds\_total**(gauge) | Total user and system CPU time spent in seconds*Shown as second* | | **streamnative.source\_connector.pulsar\_source\_last\_invocation**(gauge) | The timestamp of the last invocation of the source*Shown as millisecond* | | **streamnative.source\_connector.pulsar\_source\_received\_1min\_total**(gauge) | The total number of records received from source in the last 1 minute*Shown as record* | | **streamnative.source\_connector.pulsar\_source\_received\_total**(gauge) | The total number of records received from source*Shown as record* | | **streamnative.source\_connector.pulsar\_source\_source\_exception**(gauge) | The exception from a source | | **streamnative.source\_connector.pulsar\_source\_source\_exceptions\_1min\_total**(gauge) | The total number of source exceptions in the last 1 minute*Shown as exception* | | **streamnative.source\_connector.pulsar\_source\_source\_exceptions\_total**(gauge) | The total number of source exceptions*Shown as exception* | | **streamnative.source\_connector.pulsar\_source\_system\_exception**(gauge) | The exception from system code | | **streamnative.source\_connector.pulsar\_source\_system\_exceptions\_1min\_total**(gauge) | The total number of system exceptions in the last 1 minute*Shown as exception* | | **streamnative.source\_connector.pulsar\_source\_system\_exceptions\_total**(gauge) | The total number of system exceptions*Shown as exception* | | **streamnative.source\_connector.pulsar\_source\_written\_1min\_total**(gauge) | The total number of records written to a Pulsar topic in the last 1 minute*Shown as record* | | **streamnative.source\_connector.pulsar\_source\_written\_total**(gauge) | The total number of records written to a Pulsar topic*Shown as record* | ### Service Checks{% #service-checks %} The StreamNative integration does not include any service checks. ### Events{% #events %} The StreamNative integration does not include any events. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).