StreamNative

StreamNative - Health
StreamNative - Kafka Connect
StreamNative - Pulsar Resource
StreamNative - Sink Connector
StreamNative - Source Connector
이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Overview

StreamNative provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services.

The StreamNative integration collects the following types of metrics:

  1. Health
  2. Pulsar Resource
  3. Source Connector
  4. Sink Connector
  5. Kafka Connect

Setup

Generate API credentials in StreamNative

  1. Log into StreamNative Cloud Console Account.
  2. Click the profile icon and navigate to the Accounts & Accesses tab.
  3. Find the Service Account with Admin permissions or a binding to the metrics-viewer role.
    • If no Service Account exists, select New -> Service Account to create one, and make sure to enable the Super Admin option.
    • To bind a service account with the metrics-viewer role, refer to the metrics-viewer rolebinding documentation.
  4. On the right side of the chosen Service Account, click the ... button.
  5. Select Download OAuth2 Key to obtain the Client ID and Client Secret.

Get Organization ID and Instance Name

  1. Click the profile icon and select Organizations.
  2. Choose the Organization for which data needs to be collected.
  3. From the Select an Instance dropdown, get the Instance Name.

Connect your StreamNative account to Datadog

  1. Add your organization ID, instance name, client ID, and client secret.
    ParametersDescription
    Organization IDOrganization ID of your StreamNative account.
    Instance NameInstance name of the corresponding organization.
    Client IDClient ID of your service account.
    Client SecretClient secret of your service account.
  2. Click the Save button to save your settings.

Data Collected

Logs

The StreamNative integration does not include any logs.

Metrics

The StreamNative integration collects and forwards the following metrics to Datadog.

streamnative.health.pulsar_detector_e2e_latency_ms
(gauge)		The latency distribution from message sending to message consumption
Shown as millisecond
streamnative.health.pulsar_detector_geo_latency_ms
(gauge)		The latency distribution Latency distribution from message sending to message consumption across clusters
Shown as millisecond
streamnative.health.pulsar_detector_publish_latency_ms
(gauge)		The latency distribution of message sending
Shown as millisecond
streamnative.health.pulsar_detector_pulsar_sla_messaging_up
(gauge)		The gauge for indicating the messaging service up or down
streamnative.health.pulsar_detector_pulsar_sla_webservice_up
(gauge)		The gauge for indicating the webservice up or down
streamnative.kafka_connect.jvm_gc_collection_seconds_sum
(gauge)		Time spent in a given JVM garbage collector in seconds.
Shown as second
streamnative.kafka_connect.jvm_memory_committed_bytes
(gauge)		Committed bytes of a given JVM memory area
Shown as byte
streamnative.kafka_connect.jvm_memory_init_bytes
(gauge)		Initial bytes of a given JVM memory area
Shown as byte
streamnative.kafka_connect.jvm_memory_max_bytes
(gauge)		Max bytes of a given JVM memory area
Shown as byte
streamnative.kafka_connect.jvm_memory_used_bytes
(gauge)		Used bytes of a given JVM memory area
Shown as byte
streamnative.kafka_connect.kafka_connect_connector_task_batch_size_avg
(gauge)		The average size of the batches processed by the connector
streamnative.kafka_connect.kafka_connect_connector_task_batch_size_max
(gauge)		The maximum size of the batches processed by the connector
streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_avg_time_ms
(gauge)		The average time in milliseconds taken by this task to commit offsets
Shown as millisecond
streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_failure_percentage
(gauge)		The average percentage of this task's offset commit attempts that failed
Shown as fraction
streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_max_time_ms
(gauge)		The maximum time in milliseconds taken by this task to commit offsets
Shown as millisecond
streamnative.kafka_connect.kafka_connect_connector_task_offset_commit_success_percentage
(gauge)		The average percentage of this task's offset commit attempts that succeeded
Shown as fraction
streamnative.kafka_connect.kafka_connect_connector_task_pause_ratio
(gauge)		The fraction of time this task has spent in the pause state
Shown as fraction
streamnative.kafka_connect.kafka_connect_connector_task_running_ratio
(gauge)		The fraction of time this task has spent in the running state
Shown as fraction
streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_completion
(gauge)		The total number of offset commit completions that were completed successfully
Shown as offset
streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_completion_rate
(gauge)		The average per-second number of offset commit completions that were completed successfully
Shown as offset
streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_seq_no
(gauge)		The current sequence number for offset commits
streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_skip
(gauge)		The total number of offset commit completions that were received too late and skipped/ignored
Shown as offset
streamnative.kafka_connect.kafka_connect_sink_task_offset_commit_skip_rate
(gauge)		The average per-second number of offset commit completions that were received too late and skipped/ignored
Shown as offset
streamnative.kafka_connect.kafka_connect_sink_task_partition_count
(gauge)		The number of topic partitions assigned to this task belonging to the named sink connector in this worker
streamnative.kafka_connect.kafka_connect_sink_task_put_batch_avg_time_ms
(gauge)		The average time taken by this task to put a batch of sinks records
Shown as millisecond
streamnative.kafka_connect.kafka_connect_sink_task_put_batch_max_time_ms
(gauge)		The maximum time taken by this task to put a batch of sinks records
Shown as millisecond
streamnative.kafka_connect.kafka_connect_sink_task_sink_record_active_count
(gauge)		The number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task
Shown as record
streamnative.kafka_connect.kafka_connect_sink_task_sink_record_active_count_avg
(gauge)		The average number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task
Shown as record
streamnative.kafka_connect.kafka_connect_sink_task_sink_record_active_count_max
(gauge)		The maximum number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task
Shown as record
streamnative.kafka_connect.kafka_connect_sink_task_sink_record_read
(gauge)		The total number of records read from Kafka by this task belonging to the named sink connector in this worker, since the task was last restarted
Shown as record
streamnative.kafka_connect.kafka_connect_sink_task_sink_record_read_rate
(gauge)		The average per-second number of records read from Kafka for this task belonging to the named sink connector in this worker. This is before transformations are applied
Shown as record
streamnative.kafka_connect.kafka_connect_sink_task_sink_record_send
(gauge)		The total number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker, since the task was last restarted
Shown as record
streamnative.kafka_connect.kafka_connect_sink_task_sink_record_send_rate
(gauge)		The average per-second number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker
Shown as record
streamnative.kafka_connect.kafka_connect_source_task_poll_batch_avg_time_ms
(gauge)		The average time in milliseconds taken by this task to poll for a batch of source records
Shown as millisecond
streamnative.kafka_connect.kafka_connect_source_task_poll_batch_max_time_ms
(gauge)		The maximum time in milliseconds taken by this task to poll for a batch of source records
Shown as millisecond
streamnative.kafka_connect.kafka_connect_source_task_source_record_active_count
(gauge)		The number of records that have been produced by this task but not yet completely written to Kafka
Shown as record
streamnative.kafka_connect.kafka_connect_source_task_source_record_active_count_avg
(gauge)		The average number of records that have been produced by this task but not yet completely written to Kafka
Shown as record
streamnative.kafka_connect.kafka_connect_source_task_source_record_active_count_max
(gauge)		The maximum number of records that have been produced by this task but not yet completely written to Kafka
Shown as record
streamnative.kafka_connect.kafka_connect_source_task_source_record_poll
(gauge)		The total number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker
Shown as record
streamnative.kafka_connect.kafka_connect_source_task_source_record_poll_rate
(gauge)		The average per-second number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker
Shown as record
streamnative.kafka_connect.kafka_connect_source_task_source_record_write
(gauge)		The number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker, since the task was last restarted
Shown as record
streamnative.kafka_connect.kafka_connect_source_task_source_record_write_rate
(gauge)		The average per-second number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker
Shown as record
streamnative.kafka_connect.kafka_connect_task_error_deadletterqueue_produce_failures
(gauge)		The number of failed writes to the dead letter queue
streamnative.kafka_connect.kafka_connect_task_error_deadletterqueue_produce_requests
(gauge)		The number of attempted writes to the dead letter queue
streamnative.kafka_connect.kafka_connect_task_error_last_error_timestamp
(gauge)		The epoch timestamp when this task last encountered an error
Shown as millisecond
streamnative.kafka_connect.kafka_connect_task_error_total_errors_logged
(gauge)		The total number of errors that were logged
Shown as error
streamnative.kafka_connect.kafka_connect_task_error_total_record_errors
(gauge)		The total number of record processing errors in this task
Shown as record
streamnative.kafka_connect.kafka_connect_task_error_total_record_failures
(gauge)		The total number of record processing failures in this task
Shown as record
streamnative.kafka_connect.kafka_connect_task_error_total_records_skipped
(gauge)		The total number of records skipped due to errors
Shown as record
streamnative.kafka_connect.kafka_connect_task_error_total_retries
(gauge)		The total number of operations retried
Shown as operation
streamnative.kafka_connect.kafka_connect_worker_connector_destroyed_task_count
(gauge)		The number of destroyed tasks of the connector on the worker
Shown as task
streamnative.kafka_connect.kafka_connect_worker_connector_failed_task_count
(gauge)		The number of failed tasks of the connector on the worker
Shown as task
streamnative.kafka_connect.kafka_connect_worker_connector_paused_task_count
(gauge)		The number of paused tasks of the connector on the worker
Shown as task
streamnative.kafka_connect.kafka_connect_worker_connector_restarting_task_count
(gauge)		The number of restarting tasks of the connector on the worker
Shown as task
streamnative.kafka_connect.kafka_connect_worker_connector_running_task_count
(gauge)		The number of running tasks of the connector on the worker
Shown as task
streamnative.kafka_connect.kafka_connect_worker_connector_total_task_count
(gauge)		The number of tasks of the connector on the worker
Shown as task
streamnative.kafka_connect.kafka_connect_worker_connector_unassigned_task_count
(gauge)		The number of unassigned tasks of the connector on the worker
Shown as task
streamnative.kafka_connect.process_cpu_seconds_total
(gauge)		Total user and system CPU time spent in seconds
Shown as second
streamnative.pulsar_resource.pulsar_consumers_count
(gauge)		The number of active consumers of the topic connected to this broker.
streamnative.pulsar_resource.pulsar_entry_size_le_100_kb
(gauge)		The entry rate for a topic with entry size smaller than or equal to 100 kilobytes
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_128
(gauge)		The entry rate for a topic with entry size smaller than or equal to 128 bytes
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_16_kb
(gauge)		The entry rate for a topic with entry size smaller than or equal to 16 kilobytes
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_1_kb
(gauge)		The entry rate for a topic with entry size smaller than or equal to 1 kilobyte
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_1_mb
(gauge)		The entry rate for a topic with entry size smaller than or equal to 1 megabyte
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_2_kb
(gauge)		The entry rate for a topic with entry size smaller than or equal to 2 kilobytes
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_4_kb
(gauge)		The entry rate for a topic with entry size smaller than or equal to 4 kilobytes
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_512
(gauge)		The entry rate for a topic with entry size smaller than or equal to 512 bytes
Shown as fraction
streamnative.pulsar_resource.pulsar_entry_size_le_overflow
(gauge)		The entry rate for a topic with entry size greater than 1 megabyte
Shown as fraction
streamnative.pulsar_resource.pulsar_producers_count
(gauge)		The number of active producers of the topic connected to this broker
streamnative.pulsar_resource.pulsar_rate_in
(gauge)		The total message rate of the namespace coming into this broker
Shown as message
streamnative.pulsar_resource.pulsar_rate_out
(gauge)		The total message rate of the namespace going out from this broker
Shown as message
streamnative.pulsar_resource.pulsar_storage_backlog_size
(gauge)		The total backlog size of the topics of this topic owned by this broker
Shown as byte
streamnative.pulsar_resource.pulsar_storage_offloaded_size
(gauge)		The total amount of the data in this topic offloaded to the tiered storage
Shown as byte
streamnative.pulsar_resource.pulsar_storage_read_rate
(gauge)		The total message batches (entries) read from the storage for this topic
streamnative.pulsar_resource.pulsar_storage_size
(gauge)		The total storage size of the topics in this topic owned by this broker
Shown as byte
streamnative.pulsar_resource.pulsar_storage_write_latency_le_0_5
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 0.5 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_1
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 1 millisecond
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_10
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 10 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_100
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 100 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_1000
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 1000 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_20
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 20 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_200
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 200 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_5
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 5 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_50
(gauge)		The entry rate for a topic where the storage write latency is less than or equal to 50 milliseconds
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_latency_le_overflow
(gauge)		The entry rate for a topic where the storage write latency is greater than 1 second
Shown as fraction
streamnative.pulsar_resource.pulsar_storage_write_rate
(gauge)		The total message batches (entries) written to the storage for this topic
streamnative.pulsar_resource.pulsar_subscription_delayed
(gauge)		The total message batches (entries) are delayed for dispatching
streamnative.pulsar_resource.pulsar_subscriptions_count
(gauge)		The number of Pulsar subscriptions of the topic served by this broker
streamnative.pulsar_resource.pulsar_throughput_in
(gauge)		The total throughput of the topic coming into this broker
Shown as byte
streamnative.pulsar_resource.pulsar_throughput_out
(gauge)		The total throughput of the topic going out from this broker
Shown as byte
streamnative.pulsar_resource.pulsar_topics_count
(gauge)		The number of Pulsar topics of the namespace owned by this broker
streamnative.sink_connector.jvm_gc_collection_seconds_sum
(gauge)		Time spent in a given JVM garbage collector in seconds
Shown as second
streamnative.sink_connector.jvm_memory_bytes_committed
(gauge)		Committed bytes of a given JVM memory area
Shown as byte
streamnative.sink_connector.jvm_memory_bytes_init
(gauge)		Initial bytes of a given JVM memory area
Shown as byte
streamnative.sink_connector.jvm_memory_bytes_max
(gauge)		Max bytes of a given JVM memory area
Shown as byte
streamnative.sink_connector.jvm_memory_direct_bytes_used
(gauge)		Used bytes of a given JVM memory area
Shown as byte
streamnative.sink_connector.process_cpu_seconds_total
(gauge)		Total user and system CPU time spent in seconds
Shown as second
streamnative.sink_connector.pulsar_sink_last_invocation
(gauge)		The timestamp of the last invocation of the sink
Shown as millisecond
streamnative.sink_connector.pulsar_sink_received_1min_total
(gauge)		The total number of records received from sink in the last 1 minute
Shown as record
streamnative.sink_connector.pulsar_sink_received_total
(gauge)		The total number of records received from sink
Shown as record
streamnative.sink_connector.pulsar_sink_sink_exception
(gauge)		The exception from a sink
streamnative.sink_connector.pulsar_sink_sink_exceptions_1min_total
(gauge)		The total number of sink exceptions in the last 1 minute
Shown as exception
streamnative.sink_connector.pulsar_sink_sink_exceptions_total
(gauge)		The total number of sink exceptions
Shown as exception
streamnative.sink_connector.pulsar_sink_system_exception
(gauge)		The exception from system code
streamnative.sink_connector.pulsar_sink_system_exceptions_1min_total
(gauge)		The total number of system exceptions in the last 1 minute
Shown as exception
streamnative.sink_connector.pulsar_sink_system_exceptions_total
(gauge)		The total number of system exceptions
Shown as exception
streamnative.sink_connector.pulsar_sink_written_1min_total
(gauge)		The total number of records written to a Pulsar topic in the last 1 minute
Shown as record
streamnative.sink_connector.pulsar_sink_written_total
(gauge)		The total number of records written to a Pulsar topic
Shown as record
streamnative.source_connector.jvm_gc_collection_seconds_sum
(gauge)		Time spent in a given JVM garbage collector in seconds
Shown as second
streamnative.source_connector.jvm_memory_bytes_committed
(gauge)		Committed bytes of a given JVM memory area
Shown as byte
streamnative.source_connector.jvm_memory_bytes_init
(gauge)		Initial bytes of a given JVM memory area
Shown as byte
streamnative.source_connector.jvm_memory_bytes_max
(gauge)		Max bytes of a given JVM memory area
Shown as byte
streamnative.source_connector.jvm_memory_direct_bytes_used
(gauge)		Used bytes of a given JVM memory area
Shown as byte
streamnative.source_connector.process_cpu_seconds_total
(gauge)		Total user and system CPU time spent in seconds
Shown as second
streamnative.source_connector.pulsar_source_last_invocation
(gauge)		The timestamp of the last invocation of the source
Shown as millisecond
streamnative.source_connector.pulsar_source_received_1min_total
(gauge)		The total number of records received from source in the last 1 minute
Shown as record
streamnative.source_connector.pulsar_source_received_total
(gauge)		The total number of records received from source
Shown as record
streamnative.source_connector.pulsar_source_source_exception
(gauge)		The exception from a source
streamnative.source_connector.pulsar_source_source_exceptions_1min_total
(gauge)		The total number of source exceptions in the last 1 minute
Shown as exception
streamnative.source_connector.pulsar_source_source_exceptions_total
(gauge)		The total number of source exceptions
Shown as exception
streamnative.source_connector.pulsar_source_system_exception
(gauge)		The exception from system code
streamnative.source_connector.pulsar_source_system_exceptions_1min_total
(gauge)		The total number of system exceptions in the last 1 minute
Shown as exception
streamnative.source_connector.pulsar_source_system_exceptions_total
(gauge)		The total number of system exceptions
Shown as exception
streamnative.source_connector.pulsar_source_written_1min_total
(gauge)		The total number of records written to a Pulsar topic in the last 1 minute
Shown as record
streamnative.source_connector.pulsar_source_written_total
(gauge)		The total number of records written to a Pulsar topic
Shown as record

Service Checks

The StreamNative integration does not include any service checks.

Events

The StreamNative integration does not include any events.

Troubleshooting

Need help? Contact Datadog support.