Join us at the Dash conference! July 16-17, NYC

Runtime Metrics

Enable runtime metrics collection in the tracing client to gain additional insights into an application’s performance. Runtime metrics can be viewed in the context of a service, correlated in the Trace View at the time of a given request, and utilized anywhere in the platform.

Automatic Configuration

JVM metrics collection is enabled by default. It can be disabled with one configuration parameter in the tracing client, either through a system property, -Ddd.jmxfetch.enabled=false, or through an environment variable, DD_JMXFETCH_ENABLED=false.

JVM metrics can be viewed in correlation with your Java services. See the Service page in Datadog.

Note: For the runtime UI, dd-trace-java >= 0.24.0 is supported.

By default, runtime metrics from your application are sent to the Datadog Agent thanks to DogStatsD over port 8125. Make sure that DogStatsD is enabled for the Agent. If you are running the Agent as a container, ensure that DD_DOGSTATSD_NON_LOCAL_TRAFFIC is set to true, and that port 8125 is open on the Agent. In Kubernetes, bind the DogstatsD port to a host port; in ECS, set the appropriate flags in your task definition.

This feature is currently in private beta. Reach out to support to turn on this feature for your account.

Runtime metrics collection can be enabled with the DD_RUNTIME_METRICS_ENABLED=true environment parameter when running with ddtrace-run:

Runtime metrics can be viewed in correlation with your Python services. See the Service page in Datadog.

Note: For the runtime UI, ddtrace >= 0.24.0 is supported.

By default, runtime metrics from your application are sent to the Datadog Agent thanks to DogStatsD over port 8125. Make sure that DogStatsD is enabled for the Agent. If you are running the Agent as a container, ensure that DD_DOGSTATSD_NON_LOCAL_TRAFFIC is set to true, and that port 8125 is open on the Agent. In Kubernetes, bind the DogstatsD port to a host port; in ECS, set the appropriate flags in your task definition.

This feature is currently in private beta. Reach out to support to turn on this feature for your account.

Runtime metrics collection uses the dogstatsd-ruby gem to send metrics via DogStatsD to the Agent. To collect runtime metrics, you must add this gem to your Ruby application, and make sure that DogStatsD is enabled for the Agent.

Metrics collection is disabled by default. You can enable it by setting the DD_RUNTIME_METRICS_ENABLED environment variable to true, or by setting the following configuration in your Ruby application:

# config/initializers/datadog.rb
require 'datadog/statsd'
require 'ddtrace'

Datadog.configure do |c|
  # To enable runtime metrics collection, set `true`. Defaults to `false`
  # You can also set DD_RUNTIME_METRICS_ENABLED=true to configure this.
  c.runtime_metrics_enabled = true

  # Optionally, you can configure the DogStatsD instance used for sending runtime metrics.
  # DogStatsD is automatically configured with default settings if `dogstatsd-ruby` is available.
  # You can configure with host and port of Datadog agent; defaults to 'localhost:8125'.
  c.runtime_metrics statsd: Datadog::Statsd.new
end

Runtime metrics can be viewed in correlation with your Ruby services. See the Service page in Datadog.

By default, runtime metrics from your application are sent to the Datadog Agent thanks to DogStatsD over port 8125. Make sure that DogStatsD is enabled for the Agent. If you are running the Agent as a container, ensure that DD_DOGSTATSD_NON_LOCAL_TRAFFIC is set to true, and that port 8125 is open on the Agent. In Kubernetes, bind the DogstatsD port to a host port; in ECS, set the appropriate flags in your task definition.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

This feature is currently in private beta. Reach out to support to turn on this feature for your account.

Runtime metrics collection can be enabled with one configuration parameter in the tracing client either through the tracer option: tracer.init({ runtimeMetrics: true }) or through the environment variable: DD_RUNTIME_METRICS_ENABLED=true

Runtime metrics can be viewed in correlation with your Node services. See the Service page in Datadog.

By default, runtime metrics from your application are sent to the Datadog Agent thanks to DogStatsD over port 8125. Make sure that DogStatsD is enabled for the Agent. If you are running the Agent as a container, ensure that DD_DOGSTATSD_NON_LOCAL_TRAFFIC is set to true, and that port 8125 is open on the Agent. In Kubernetes, bind the DogstatsD port to a host port; in ECS, set the appropriate flags in your task definition.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Data Collected

The following metrics are collected by default after enabling JVM metrics.

jvm.heap_memory
(gauge)
The total Java heap memory used.
shown as byte
jvm.heap_memory_committed
(gauge)
The total Java heap memory committed to be used.
shown as byte
jvm.heap_memory_init
(gauge)
The initial Java heap memory allocated.
shown as byte
jvm.heap_memory_max
(gauge)
The maximum Java heap memory available.
shown as byte
jvm.non_heap_memory
(gauge)
The total Java non-heap memory used.
shown as byte
jvm.non_heap_memory_committed
(gauge)
The total Java non-heap memory committed to be used.
shown as byte
jvm.non_heap_memory_init
(gauge)
The initial Java non-heap memory allocated.
shown as byte
jvm.non_heap_memory_max
(gauge)
The maximum Java non-heap memory available.
shown as byte
jvm.thread_count
(gauge)
The number of live threads.
shown as thread
jvm.gc.cms.count
(gauge)
The total number of garbage collections that have occurred.
jvm.gc.parnew.time
(gauge)
The approximate accumulated garbage collection time elapsed.
shown as millisecond

Along with displaying these metrics in your APM Service Page, Datadog provides a default JVM Runtime Dashboard with the service and runtime-id tags that are applied to these metrics.

Additional JMX metrics can be added using configuration files that are passed to jmxfetch.metrics-configs. You can also enable existing Datadog JMX integrations individually with the dd.integration.<name> parameter. This auto-embeds configuration from Datadog’s existing JMX configuration files. See the JMX Integration for further details on configuration.

runtime.python.cpu.time.sys
(gauge)
Number of seconds executing in the kernel
shown as second
runtime.python.cpu.time.user
(gauge)
Number of seconds executing outside the kernel
shown as second
runtime.python.cpu.percent
(gauge)
CPU utilization percentage
shown as percent
runtime.python.cpu.ctx_switch.voluntary
(gauge)
Number of voluntary context switches
shown as invocation
runtime.python.cpu.ctx_switch.involuntary
(gauge)
Number of involuntary context switches
shown as invocation
runtime.python.gc.count.gen0
(gauge)
Number of generation 0 objects
shown as resource
runtime.python.gc.count.gen1
(gauge)
Number of generation 1 objects
shown as resource
runtime.python.gc.count.gen2
(gauge)
Number of generation 2 objects
shown as resource
runtime.python.mem.rss
(gauge)
Resident set memory
shown as byte
runtime.python.thread_count
(gauge)
Number of threads
shown as thread

Along with displaying these metrics in your APM Service Page, Datadog provides a default Python Runtime Metrics Dashboard with the service and runtime-id tags that are applied to these metrics.

The following metrics are collected by default after enabling Runtime metrics.

runtime.ruby.class_count
(gauge)
Total number of classes loaded
shown as resource
runtime.ruby.gc.remembered_wb_unprotected_objects
(gauge)
Number of write-barrier unprotected objects in the remembered set
shown as resource
runtime.ruby.gc.remembered_wb_unprotected_objects_limit
(gauge)
Limit on write-barrier unprotected objects allowed in the remembered set
shown as resource
runtime.ruby.gc.oldmalloc_increase_bytes
(gauge)
Total bytes allocated to old objects
shown as byte
runtime.ruby.gc.oldmalloc_increase_bytes_limit
(gauge)
Bytes limit that will trigger garbage collection of old objects
shown as byte
runtime.ruby.gc.malloc_increase_bytes
(gauge)
Total bytes allocated to objects
shown as byte
runtime.ruby.gc.malloc_increase_bytes_limit
(gauge)
Bytes limit that will trigger garbage collection of objects
shown as byte
runtime.ruby.gc.total_allocated_objects
(gauge)
Total number of allocated objects over the lifetime of this process
shown as resource
runtime.ruby.gc.total_freed_objects
(gauge)
Total number of freed objects over the lifetime of this process
shown as resource
runtime.ruby.gc.total_allocated_pages
(gauge)
Total number of allocated pages over the lifetime of this process
shown as page
runtime.ruby.gc.total_freed_pages
(gauge)
Total number of freed pages over the lifetime of this process
shown as page
runtime.ruby.gc.heap_live_slots
(gauge)
Number of live objects slots
shown as resource
runtime.ruby.gc.heap_final_slots
(gauge)
Number of object slots with finalizers attached to them
shown as resource
runtime.ruby.gc.heap_marked_slots
(gauge)
Count of old objects which survived more than 3 GC cycles and number of write-barrier unprotected objects
shown as resource
runtime.ruby.gc.heap_available_slots
(gauge)
Total number of slots in heap pages
shown as resource
runtime.ruby.gc.heap_free_slots
(gauge)
Number of free slots in heap pages
shown as resource
runtime.ruby.thread_count
(gauge)
Total number of threads
shown as thread
runtime.ruby.gc.old_objects
(gauge)
Total number of old objects
shown as resource
runtime.ruby.gc.old_objects_limit
(gauge)
Limit on number of old objects
shown as resource

Along with displaying these metrics in your APM Service Page, Datadog provides a default Ruby Runtime Dashboard with the service and runtime-id tags that are applied to these metrics.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

The following metrics are collected by default after enabling runtime metrics.

runtime.node.cpu.user
(gauge)
CPU usage in user code
shown as percent
runtime.node.cpu.system
(gauge)
CPU usage in system code
shown as percent
runtime.node.cpu.total
(gauge)
Total CPU usage
shown as percent
runtime.node.mem.rss
(gauge)
Resident set size
shown as byte
runtime.node.mem.heap_total
(gauge)
Total heap memory
shown as byte
runtime.node.mem.heap_used
(gauge)
Heap memory usage
shown as byte
runtime.node.mem.external
(gauge)
External memory
shown as byte
runtime.node.heap.total_heap_size
(gauge)
Total heap size
shown as byte
runtime.node.heap.total_heap_size_executable
(gauge)
Total executable heap size
shown as byte
runtime.node.heap.total_physical_size
(gauge)
Total physical heap size
shown as byte
runtime.node.heap.used_heap_size
(gauge)
Used heap size
shown as byte
runtime.node.heap.heap_size_limit
(gauge)
Heap size limit
shown as byte
runtime.node.heap.malloced_memory
(gauge)
Malloced memory
shown as byte
runtime.node.heap.peak_malloced_memory
(gauge)
Peak allocated memory
shown as byte
runtime.node.heap.size.by.space
(gauge)
Heap space size
shown as byte
runtime.node.heap.used_size.by.space
(gauge)
Heap space used size
shown as byte
runtime.node.heap.available_size.by.space
(gauge)
Heap space available size
shown as byte
runtime.node.heap.physical_size.by.space
(gauge)
Heap space physical size
shown as byte
runtime.node.process.uptime
(gauge)
Process uptime
shown as second
runtime.node.event_loop.delay.max
(gauge)
Maximum event loop delay
shown as nanosecond
runtime.node.event_loop.delay.min
(gauge)
Minimum event loop delay
shown as nanosecond
runtime.node.event_loop.delay.avg
(gauge)
Average event loop delay
shown as nanosecond
runtime.node.event_loop.delay.sum
(rate)
Total event loop delay
shown as nanosecond
runtime.node.event_loop.delay.median
(gauge)
Median event loop delay
shown as nanosecond
runtime.node.event_loop.delay.95percentile
(gauge)
95th percentile event loop delay
shown as nanosecond
runtime.node.event_loop.delay.count
(rate)
Event loop iterations count
shown as execution
runtime.node.gc.pause.max
(gauge)
Maximum garbage collection pause
shown as nanosecond
runtime.node.gc.pause.min
(gauge)
Minimum garbage collection pause
shown as nanosecond
runtime.node.gc.pause.avg
(gauge)
Average garbage collection pause
shown as nanosecond
runtime.node.gc.pause.sum
(rate)
Total garbage collection pause
shown as nanosecond
runtime.node.gc.pause.median
(gauge)
Median garbage collection pause
shown as nanosecond
runtime.node.gc.pause.95percentile
(gauge)
95th percentile garbage collection pause
shown as nanosecond
runtime.node.gc.pause.count
(rate)
Number of garbage collections
shown as garbage collection
runtime.node.gc.pause.by.type.max
(gauge)
Maximum garbage collection pause by type
shown as nanosecond
runtime.node.gc.pause.by.type.min
(gauge)
Minimum garbage collection pause by type
shown as nanosecond
runtime.node.gc.pause.by.type.avg
(gauge)
Average garbage collection pause by type
shown as nanosecond
runtime.node.gc.pause.by.type.sum
(rate)
Total garbage collection pause by type
shown as nanosecond
runtime.node.gc.pause.by.type.median
(gauge)
Median garbage collection pause by type
shown as nanosecond
runtime.node.gc.pause.by.type.95percentile
(gauge)
95th percentile garbage collection pause by type
shown as nanosecond
runtime.node.gc.pause.by.type.count
(rate)
Number of garbage collections by type
shown as garbage collection
runtime.node.spans.finished
(gauge)
Number of finished spans held in memory
shown as instance
runtime.node.spans.finished.by.name
(gauge)
Number of finished spans held in memory by span name
shown as instance
runtime.node.spans.unfinished
(gauge)
Number of unfinished spans held in memory
shown as instance
runtime.node.spans.unfinished.by.name
(gauge)
Number of unfinished spans held in memory by span name
shown as instance
runtime.node.async.resources
(gauge)
Number of asynchronous resources held in memory
shown as resource
runtime.node.async.resources.by.type
(gauge)
Number of asynchronous resources held in memory
shown as resource

Along with displaying these metrics in your APM Service Page, Datadog provides a default Node Runtime Dashboard with the service and runtime-id tags that are applied to these metrics.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Further Reading