--- title: StreamNative description: Gain insights into StreamNative metrics data. breadcrumbs: Docs > Integrations > StreamNative --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # StreamNative Integration version1.0.0 {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} StreamNative - HealthStreamNative - Kafka ConnectStreamNative - Pulsar ResourceStreamNative - Sink ConnectorStreamNative - Source Connector ## Overview{% #overview %} [StreamNative](https://streamnative.io/) provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services. The StreamNative integration collects the following types of [metrics](https://docs.streamnative.io/docs/cloud-metrics-api#metrics-endpoint): 1. Health 1. Pulsar Resource 1. Source Connector 1. Sink Connector 1. Kafka Connect ## Setup{% #setup %} ### Generate API credentials in StreamNative{% #generate-api-credentials-in-streamnative %} 1. Log into [StreamNative Cloud Console Account](https://console.streamnative.cloud/). 1. Click the profile icon and navigate to the **Accounts & Accesses** tab. 1. Find the Service Account with **Admin** permissions or a binding to the `metrics-viewer` role. - If no Service Account exists, select **New -> Service Account** to create one, and make sure to enable the **Super Admin** option. - To bind a service account with the `metrics-viewer` role, refer to the [metrics-viewer rolebinding](https://docs.streamnative.io/docs/cloud-metrics-api#metrics-authorization) documentation. 1. On the right side of the chosen Service Account, click the `...` button. 1. Select **Download OAuth2 Key** to obtain the **Client ID** and **Client Secret**. ### Get `Organization ID` and `Instance Name`{% #get-organization-id-and-instance-name %} 1. Click the profile icon and select **Organizations**. 1. Choose the **Organization** for which data needs to be collected. 1. From the **Select an Instance** dropdown, get the **Instance Name**. ### Connect your StreamNative account to Datadog{% #connect-your-streamnative-account-to-datadog %} 1. Add your organization ID, instance name, client ID, and client secret. | Parameters | Description | | --------------- | ------------------------------------------------ | | Organization ID | Organization ID of your StreamNative account. | | Instance Name | Instance name of the corresponding organization. | | Client ID | Client ID of your service account. | | Client Secret | Client secret of your service account. | 1. Click the **Save** button to save your settings. ## Data Collected{% #data-collected %} ### Logs{% #logs %} The StreamNative integration does not include any logs. ### Metrics{% #metrics %} | | | | | **streamnative.health.pulsar_detector_e2e_latency_ms**(gauge) | The latency distribution from message sending to message consumption*Shown as millisecond* | | **streamnative.health.pulsar_detector_geo_latency_ms**(gauge) | The latency distribution Latency distribution from message sending to message consumption across clusters*Shown as millisecond* | | **streamnative.health.pulsar_detector_publish_latency_ms**(gauge) | The latency distribution of message sending*Shown as millisecond* | | **streamnative.health.pulsar_detector_pulsar_sla_messaging_up**(gauge) | The gauge for indicating the messaging service up or down | | **streamnative.health.pulsar_detector_pulsar_sla_webservice_up**(gauge) | The gauge for indicating the webservice up or down | | **streamnative.kafka_connect.jvm_gc_collection_seconds_sum**(gauge) | Time spent in a given JVM garbage collector in seconds.*Shown as second* | | **streamnative.kafka_connect.jvm_memory_committed_bytes**(gauge) | Committed bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka_connect.jvm_memory_init_bytes**(gauge) | Initial bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka_connect.jvm_memory_max_bytes**(gauge) | Max bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka_connect.jvm_memory_used_bytes**(gauge) | Used bytes of a given JVM memory area*Shown as byte* | | **streamnative.kafka_connect.kafka_connect_connector_task_batch_size_avg**(gauge) | The average size of the batches processed by the connector | | **streamnative.kafka_connect.kafka_connect_connector_task_batch_size_max**(gauge) | The maximum size of the batches processed by the connector | | **streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_avg_time_ms**(gauge) | The average time in milliseconds taken by this task to commit offsets*Shown as millisecond* | | **streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_failure_percentage**(gauge) | The average percentage of this task's offset commit attempts that failed*Shown as fraction* | | **streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_max_time_ms**(gauge) | The maximum time in milliseconds taken by this task to commit offsets*Shown as millisecond* | | **streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_success_percentage**(gauge) | The average percentage of this task's offset commit attempts that succeeded*Shown as fraction* | | **streamnative.kafka_connect.kafka_connect_connector_task_pause_ratio**(gauge) | The fraction of time this task has spent in the pause state*Shown as fraction* | | **streamnative.kafka_connect.kafka_connect_connector_task_running_ratio**(gauge) | The fraction of time this task has spent in the running state*Shown as fraction* | | **streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_completion**(gauge) | The total number of offset commit completions that were completed successfully*Shown as offset* | | **streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_completion_rate**(gauge) | The average per-second number of offset commit completions that were completed successfully*Shown as offset* | | **streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_seq_no**(gauge) | The current sequence number for offset commits | | **streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_skip**(gauge) | The total number of offset commit completions that were received too late and skipped/ignored*Shown as offset* | | **streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_skip_rate**(gauge) | The average per-second number of offset commit completions that were received too late and skipped/ignored*Shown as offset* | | **streamnative.kafka_connect.kafka_connect_sink_task_partition_count**(gauge) | The number of topic partitions assigned to this task belonging to the named sink connector in this worker | | **streamnative.kafka_connect.kafka_connect_sink_task_put_batch_avg_time_ms**(gauge) | The average time taken by this task to put a batch of sinks records*Shown as millisecond* | | **streamnative.kafka_connect.kafka_connect_sink_task_put_batch_max_time_ms**(gauge) | The maximum time taken by this task to put a batch of sinks records*Shown as millisecond* | | **streamnative.kafka_connect.kafka_connect_sink_task_sink_record_active_count**(gauge) | The number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task*Shown as record* | | **streamnative.kafka_connect.kafka_connect_sink_task_sink_record_active_count_avg**(gauge) | The average number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task*Shown as record* | | **streamnative.kafka_connect.kafka_connect_sink_task_sink_record_active_count_max**(gauge) | The maximum number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task*Shown as record* | | **streamnative.kafka_connect.kafka_connect_sink_task_sink_record_read**(gauge) | The total number of records read from Kafka by this task belonging to the named sink connector in this worker, since the task was last restarted*Shown as record* | | **streamnative.kafka_connect.kafka_connect_sink_task_sink_record_read_rate**(gauge) | The average per-second number of records read from Kafka for this task belonging to the named sink connector in this worker. This is before transformations are applied*Shown as record* | | **streamnative.kafka_connect.kafka_connect_sink_task_sink_record_send**(gauge) | The total number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker, since the task was last restarted*Shown as record* | | **streamnative.kafka_connect.kafka_connect_sink_task_sink_record_send_rate**(gauge) | The average per-second number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker*Shown as record* | | **streamnative.kafka_connect.kafka_connect_source_task_poll_batch_avg_time_ms**(gauge) | The average time in milliseconds taken by this task to poll for a batch of source records*Shown as millisecond* | | **streamnative.kafka_connect.kafka_connect_source_task_poll_batch_max_time_ms**(gauge) | The maximum time in milliseconds taken by this task to poll for a batch of source records*Shown as millisecond* | | **streamnative.kafka_connect.kafka_connect_source_task_source_record_active_count**(gauge) | The number of records that have been produced by this task but not yet completely written to Kafka*Shown as record* | | **streamnative.kafka_connect.kafka_connect_source_task_source_record_active_count_avg**(gauge) | The average number of records that have been produced by this task but not yet completely written to Kafka*Shown as record* | | **streamnative.kafka_connect.kafka_connect_source_task_source_record_active_count_max**(gauge) | The maximum number of records that have been produced by this task but not yet completely written to Kafka*Shown as record* | | **streamnative.kafka_connect.kafka_connect_source_task_source_record_poll**(gauge) | The total number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker*Shown as record* | | **streamnative.kafka_connect.kafka_connect_source_task_source_record_poll_rate**(gauge) | The average per-second number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker*Shown as record* | | **streamnative.kafka_connect.kafka_connect_source_task_source_record_write**(gauge) | The number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker, since the task was last restarted*Shown as record* | | **streamnative.kafka_connect.kafka_connect_source_task_source_record_write_rate**(gauge) | The average per-second number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker*Shown as record* | | **streamnative.kafka_connect.kafka_connect_task_error_deadletterqueue_produce_failures**(gauge) | The number of failed writes to the dead letter queue | | **streamnative.kafka_connect.kafka_connect_task_error_deadletterqueue_produce_requests**(gauge) | The number of attempted writes to the dead letter queue | | **streamnative.kafka_connect.kafka_connect_task_error_last_error_timestamp**(gauge) | The epoch timestamp when this task last encountered an error*Shown as millisecond* | | **streamnative.kafka_connect.kafka_connect_task_error_total_errors_logged**(gauge) | The total number of errors that were logged*Shown as error* | | **streamnative.kafka_connect.kafka_connect_task_error_total_record_errors**(gauge) | The total number of record processing errors in this task*Shown as record* | | **streamnative.kafka_connect.kafka_connect_task_error_total_record_failures**(gauge) | The total number of record processing failures in this task*Shown as record* | | **streamnative.kafka_connect.kafka_connect_task_error_total_records_skipped**(gauge) | The total number of records skipped due to errors*Shown as record* | | **streamnative.kafka_connect.kafka_connect_task_error_total_retries**(gauge) | The total number of operations retried*Shown as operation* | | **streamnative.kafka_connect.kafka_connect_worker_connector_destroyed_task_count**(gauge) | The number of destroyed tasks of the connector on the worker*Shown as task* | | **streamnative.kafka_connect.kafka_connect_worker_connector_failed_task_count**(gauge) | The number of failed tasks of the connector on the worker*Shown as task* | | **streamnative.kafka_connect.kafka_connect_worker_connector_paused_task_count**(gauge) | The number of paused tasks of the connector on the worker*Shown as task* | | **streamnative.kafka_connect.kafka_connect_worker_connector_restarting_task_count**(gauge) | The number of restarting tasks of the connector on the worker*Shown as task* | | **streamnative.kafka_connect.kafka_connect_worker_connector_running_task_count**(gauge) | The number of running tasks of the connector on the worker*Shown as task* | | **streamnative.kafka_connect.kafka_connect_worker_connector_total_task_count**(gauge) | The number of tasks of the connector on the worker*Shown as task* | | **streamnative.kafka_connect.kafka_connect_worker_connector_unassigned_task_count**(gauge) | The number of unassigned tasks of the connector on the worker*Shown as task* | | **streamnative.kafka_connect.process_cpu_seconds_total**(gauge) | Total user and system CPU time spent in seconds*Shown as second* | | **streamnative.pulsar_resource.pulsar_consumers_count**(gauge) | The number of active consumers of the topic connected to this broker. | | **streamnative.pulsar_resource.pulsar_entry_size_le_100_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 100 kilobytes*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_128**(gauge) | The entry rate for a topic with entry size smaller than or equal to 128 bytes*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_16_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 16 kilobytes*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_1_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 1 kilobyte*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_1_mb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 1 megabyte*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_2_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 2 kilobytes*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_4_kb**(gauge) | The entry rate for a topic with entry size smaller than or equal to 4 kilobytes*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_512**(gauge) | The entry rate for a topic with entry size smaller than or equal to 512 bytes*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_entry_size_le_overflow**(gauge) | The entry rate for a topic with entry size greater than 1 megabyte*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_producers_count**(gauge) | The number of active producers of the topic connected to this broker | | **streamnative.pulsar_resource.pulsar_rate_in**(gauge) | The total message rate of the namespace coming into this broker*Shown as message* | | **streamnative.pulsar_resource.pulsar_rate_out**(gauge) | The total message rate of the namespace going out from this broker*Shown as message* | | **streamnative.pulsar_resource.pulsar_storage_backlog_size**(gauge) | The total backlog size of the topics of this topic owned by this broker*Shown as byte* | | **streamnative.pulsar_resource.pulsar_storage_offloaded_size**(gauge) | The total amount of the data in this topic offloaded to the tiered storage*Shown as byte* | | **streamnative.pulsar_resource.pulsar_storage_read_rate**(gauge) | The total message batches (entries) read from the storage for this topic | | **streamnative.pulsar_resource.pulsar_storage_size**(gauge) | The total storage size of the topics in this topic owned by this broker*Shown as byte* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_0_5**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 0.5 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_1**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 1 millisecond*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_10**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 10 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_100**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 100 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_1000**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 1000 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_20**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 20 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_200**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 200 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_5**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 5 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_50**(gauge) | The entry rate for a topic where the storage write latency is less than or equal to 50 milliseconds*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_latency_le_overflow**(gauge) | The entry rate for a topic where the storage write latency is greater than 1 second*Shown as fraction* | | **streamnative.pulsar_resource.pulsar_storage_write_rate**(gauge) | The total message batches (entries) written to the storage for this topic | | **streamnative.pulsar_resource.pulsar_subscription_delayed**(gauge) | The total message batches (entries) are delayed for dispatching | | **streamnative.pulsar_resource.pulsar_subscriptions_count**(gauge) | The number of Pulsar subscriptions of the topic served by this broker | | **streamnative.pulsar_resource.pulsar_throughput_in**(gauge) | The total throughput of the topic coming into this broker*Shown as byte* | | **streamnative.pulsar_resource.pulsar_throughput_out**(gauge) | The total throughput of the topic going out from this broker*Shown as byte* | | **streamnative.pulsar_resource.pulsar_topics_count**(gauge) | The number of Pulsar topics of the namespace owned by this broker | | **streamnative.sink_connector.jvm_gc_collection_seconds_sum**(gauge) | Time spent in a given JVM garbage collector in seconds*Shown as second* | | **streamnative.sink_connector.jvm_memory_bytes_committed**(gauge) | Committed bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink_connector.jvm_memory_bytes_init**(gauge) | Initial bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink_connector.jvm_memory_bytes_max**(gauge) | Max bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink_connector.jvm_memory_direct_bytes_used**(gauge) | Used bytes of a given JVM memory area*Shown as byte* | | **streamnative.sink_connector.process_cpu_seconds_total**(gauge) | Total user and system CPU time spent in seconds*Shown as second* | | **streamnative.sink_connector.pulsar_sink_last_invocation**(gauge) | The timestamp of the last invocation of the sink*Shown as millisecond* | | **streamnative.sink_connector.pulsar_sink_received_1min_total**(gauge) | The total number of records received from sink in the last 1 minute*Shown as record* | | **streamnative.sink_connector.pulsar_sink_received_total**(gauge) | The total number of records received from sink*Shown as record* | | **streamnative.sink_connector.pulsar_sink_sink_exception**(gauge) | The exception from a sink | | **streamnative.sink_connector.pulsar_sink_sink_exceptions_1min_total**(gauge) | The total number of sink exceptions in the last 1 minute*Shown as exception* | | **streamnative.sink_connector.pulsar_sink_sink_exceptions_total**(gauge) | The total number of sink exceptions*Shown as exception* | | **streamnative.sink_connector.pulsar_sink_system_exception**(gauge) | The exception from system code | | **streamnative.sink_connector.pulsar_sink_system_exceptions_1min_total**(gauge) | The total number of system exceptions in the last 1 minute*Shown as exception* | | **streamnative.sink_connector.pulsar_sink_system_exceptions_total**(gauge) | The total number of system exceptions*Shown as exception* | | **streamnative.sink_connector.pulsar_sink_written_1min_total**(gauge) | The total number of records written to a Pulsar topic in the last 1 minute*Shown as record* | | **streamnative.sink_connector.pulsar_sink_written_total**(gauge) | The total number of records written to a Pulsar topic*Shown as record* | | **streamnative.source_connector.jvm_gc_collection_seconds_sum**(gauge) | Time spent in a given JVM garbage collector in seconds*Shown as second* | | **streamnative.source_connector.jvm_memory_bytes_committed**(gauge) | Committed bytes of a given JVM memory area*Shown as byte* | | **streamnative.source_connector.jvm_memory_bytes_init**(gauge) | Initial bytes of a given JVM memory area*Shown as byte* | | **streamnative.source_connector.jvm_memory_bytes_max**(gauge) | Max bytes of a given JVM memory area*Shown as byte* | | **streamnative.source_connector.jvm_memory_direct_bytes_used**(gauge) | Used bytes of a given JVM memory area*Shown as byte* | | **streamnative.source_connector.process_cpu_seconds_total**(gauge) | Total user and system CPU time spent in seconds*Shown as second* | | **streamnative.source_connector.pulsar_source_last_invocation**(gauge) | The timestamp of the last invocation of the source*Shown as millisecond* | | **streamnative.source_connector.pulsar_source_received_1min_total**(gauge) | The total number of records received from source in the last 1 minute*Shown as record* | | **streamnative.source_connector.pulsar_source_received_total**(gauge) | The total number of records received from source*Shown as record* | | **streamnative.source_connector.pulsar_source_source_exception**(gauge) | The exception from a source | | **streamnative.source_connector.pulsar_source_source_exceptions_1min_total**(gauge) | The total number of source exceptions in the last 1 minute*Shown as exception* | | **streamnative.source_connector.pulsar_source_source_exceptions_total**(gauge) | The total number of source exceptions*Shown as exception* | | **streamnative.source_connector.pulsar_source_system_exception**(gauge) | The exception from system code | | **streamnative.source_connector.pulsar_source_system_exceptions_1min_total**(gauge) | The total number of system exceptions in the last 1 minute*Shown as exception* | | **streamnative.source_connector.pulsar_source_system_exceptions_total**(gauge) | The total number of system exceptions*Shown as exception* | | **streamnative.source_connector.pulsar_source_written_1min_total**(gauge) | The total number of records written to a Pulsar topic in the last 1 minute*Shown as record* | | **streamnative.source_connector.pulsar_source_written_total**(gauge) | The total number of records written to a Pulsar topic*Shown as record* | ### Service Checks{% #service-checks %} The StreamNative integration does not include any service checks. ### Events{% #events %} The StreamNative integration does not include any events. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).