--- title: OpenTelemetry Runtime Metrics description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > OpenTelemetry in Datadog > Integrations > OpenTelemetry Runtime Metrics --- # OpenTelemetry Runtime Metrics ## Overview{% #overview %} Runtime metrics provide insights into application performance, including memory usage, garbage collection, and parallelization. Datadog SDKs offer [runtime metrics collection](https://docs.datadoghq.com/tracing/metrics/runtime_metrics.md) for each supported language, and OpenTelemetry (OTel) also collects compatible runtime metrics that can be sent to Datadog through the OpenTelemetry SDKs. ## Compatibility{% #compatibility %} Datadog supports OpenTelemetry runtime metrics for the following languages: - Java - .NET - Go - Node.js - Python For details about host and container metrics mapping, see [OpenTelemetry Metrics Mapping](https://docs.datadoghq.com/opentelemetry/mapping/metrics_mapping.md). ## Setup instructions{% #setup-instructions %} ### 1. Prerequisites - You have successfully [configured OpenTelemetry metrics to send to Datadog](https://docs.datadoghq.com/opentelemetry/setup.md). - You have installed the [corresponding language integration in Datadog](https://app.datadoghq.com/integrations). ### 2. Configure your application Select your language to see instructions for configuring the OpenTelemetry SDK to send runtime metrics: **Note**: Runtime metrics are only exported if a `MeterProvider` and metric exporter are configured. Set the `OTEL_METRICS_EXPORTER` environment variable or programmatically configure a `metricReader` in your SDK initialization. For Go, Node.js, and Python, the MeterProvider must be configured manually; Java and .NET auto-configure it via their auto-instrumentation agents. {% tab title="Java" %} #### Automatic instrumentation{% #automatic-instrumentation %} If you use [OpenTelemetry automatic instrumentation](https://opentelemetry.io/docs/instrumentation/java/automatic/) for Java applications, runtime metrics are enabled by default. #### Manual instrumentation{% #manual-instrumentation %} If you use [OpenTelemetry manual instrumentation](https://opentelemetry.io/docs/instrumentation/java/manual/), follow the guides for your Java version: - [Java 8](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/runtime-telemetry/runtime-telemetry-java8/library) - [Java 17](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/runtime-telemetry/runtime-telemetry-java17/library) #### Experimental telemetry{% #experimental-telemetry %} Enabling experimental OpenTelemetry runtime telemetry may provide additional metric mappings between Datadog and OpenTelemetry for Java. To enable it, set the following environment variable: ``` OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_EMIT_EXPERIMENTAL_TELEMETRY=true ``` #### Additional JMX metrics{% #additional-jmx-metrics %} To collect additional JVM metrics beyond the default runtime instrumentation, install the [OpenTelemetry JMX Metric Scraper](https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/jmx-scraper). The scraper scrapes MBeans over JMX and emits them as OpenTelemetry metrics, which Datadog then maps to runtime metrics (see the JVM Contrib table in Data collected). Configure the scraper with `otel.jmx.target.source=legacy` to collect these additional metrics. The scraper's `instrumentation` source emits the same semantic-convention metrics already produced natively by the OpenTelemetry Java SDK, so it does not provide additional coverage. #### Collector configuration{% #collector-configuration %} The `jvm.gc.collections.count` and `jvm.gc.collections.elapsed` metrics require the [Delta-to-Rate Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatorateprocessor) in the OpenTelemetry Collector. Set `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta` or use the `cumulativetodelta` processor. ```yaml processors: deltatorate: metrics: - jvm.gc.collections.count - jvm.gc.collections.elapsed ``` **Note**: The minimum supported version of the OpenTelemetry Java agent is [2.0.0](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0). {% /tab %} {% tab title="Go" %} #### Manual instrumentation{% #manual-instrumentation %} OpenTelemetry Go applications are [instrumented manually](https://opentelemetry.io/docs/instrumentation/go/manual/). To enable runtime metrics, see the documentation for the [runtime package](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime). #### Collector configuration{% #collector-configuration %} Some Go runtime metrics are reported as monotonic sums, but Datadog expects them as gauges. The [Transform Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) converts these sums to gauges, and the [Cumulative-to-Delta Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor) excludes them from delta conversion. Verify that `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` is not set to `delta`. Otherwise, metrics excluded by `cumulativetodelta` are ignored. ```yaml processors: transform/go-runtime: error_mode: ignore metric_statements: - context: metric statements: - convert_sum_to_gauge() where name == "process.runtime.go.gc.pause_total_ns" or name == "process.runtime.go.gc.count" or name == "go.memory.allocated" or name == "go.memory.allocations" cumulativetodelta: exclude: metrics: - process.runtime.go.gc.pause_total_ns - process.runtime.go.gc.count - go.memory.allocated - go.memory.allocations match_type: strict ``` **Note**: The minimum supported version of [`go.opentelemetry.io/contrib/instrumentation/runtime`](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime) is v0.46.0, which also requires Go 1.20+ and [OpenTelemetry Go SDK v1.21.0+](https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.21.0). {% /tab %} {% tab title=".NET" %} #### Automatic instrumentation{% #automatic-instrumentation %} If you use [OpenTelemetry automatic instrumentation](https://opentelemetry.io/docs/instrumentation/net/automatic/) for .NET applications, runtime metrics are enabled by default. #### Manual instrumentation{% #manual-instrumentation %} If you use [OpenTelemetry manual instrumentation](https://opentelemetry.io/docs/instrumentation/net/manual/), see the documentation for the [OpenTelemetry.Instrumentation.Runtime library](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Runtime). #### Metric export interval{% #metric-export-interval %} The default metric export interval for the .NET OTel SDK differs from the default for the Datadog .NET SDK. Datadog recommends setting the [OTEL_METRIC_EXPORT_INTERVAL](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader) environment variable on your .NET service to match the default Datadog metric export interval: ``` OTEL_METRIC_EXPORT_INTERVAL=10000 ``` #### Collector configuration{% #collector-configuration %} The `dotnet.process.cpu.time` and `process.cpu.time` metrics require the [Delta-to-Rate Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatorateprocessor) in the OpenTelemetry Collector. Set `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta` or use the `cumulativetodelta` processor. ```yaml processors: deltatorate: metrics: - dotnet.process.cpu.time - process.cpu.time ``` **Note**: The minimum supported version of the .NET OpenTelemetry SDK is [1.5.0](https://github.com/open-telemetry/opentelemetry-dotnet/releases/tag/core-1.5.0). {% /tab %} {% tab title="Node.js" %} #### Automatic instrumentation{% #automatic-instrumentation %} If you use [OpenTelemetry automatic instrumentation](https://opentelemetry.io/docs/zero-code/js/) for Node.js applications, runtime metrics are enabled by default through the [`@opentelemetry/instrumentation-runtime-node`](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/instrumentation-runtime-node) package. #### Manual instrumentation{% #manual-instrumentation %} If you use [OpenTelemetry manual instrumentation](https://opentelemetry.io/docs/languages/js/instrumentation/), see the documentation for the [`@opentelemetry/instrumentation-runtime-node`](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/instrumentation-runtime-node) library. #### Host metrics{% #host-metrics %} Host-level metrics such as system CPU and memory usage are not included in OpenTelemetry automatic instrumentation. To collect these metrics: 1. Install and configure the [`@opentelemetry/host-metrics`](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/host-metrics) package: ```shell npm install @opentelemetry/host-metrics ``` 1. Initialize the package with your existing `MeterProvider`: ```javascript const { HostMetrics } = require('@opentelemetry/host-metrics'); const { metrics } = require('@opentelemetry/api'); const hostMetrics = new HostMetrics({ meterProvider: metrics.getMeterProvider(), }); hostMetrics.start(); ``` For the list of metrics collected by this package, see the Node.js Contrib Host table. **Note**: The minimum supported version of [`@opentelemetry/instrumentation-runtime-node`](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/instrumentation-runtime-node) is [0.9.0](https://github.com/open-telemetry/opentelemetry-js-contrib/releases/tag/instrumentation-runtime-node-v0.9.0). {% /tab %} {% tab title="Python" %} #### Installation{% #installation %} Runtime metrics are not enabled by default for Python applications. Install the [`opentelemetry-instrumentation-system-metrics`](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-system-metrics) package: ```shell pip install opentelemetry-instrumentation-system-metrics ``` #### Automatic instrumentation{% #automatic-instrumentation %} If you use [OpenTelemetry automatic instrumentation](https://opentelemetry.io/docs/zero-code/python/) for Python applications, `opentelemetry-instrument` discovers and enables the package after installation. #### Manual instrumentation{% #manual-instrumentation %} If you use [OpenTelemetry manual instrumentation](https://opentelemetry.io/docs/languages/python/instrumentation/), enable the package in your application: ```python from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor SystemMetricsInstrumentor().instrument() ``` {% /tab %} ## View runtime metric dashboards{% #view-runtime-metric-dashboards %} After setup is complete, you can view runtime metrics in: - The service's details page (see Java example below) - The flame graph metrics tab - Default runtime dashboards ([Java Example](https://app.datadoghq.com/dash/integration/256/jvm-metrics)) {% image source="https://docs.dd-static.net/images/opentelemetry/otel_runtime_metrics_service_page.684ee9c34c52d62f70b570d67b94e32b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/opentelemetry/otel_runtime_metrics_service_page.684ee9c34c52d62f70b570d67b94e32b.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Service page showing OpenTelemetry runtime metrics on the JVM Metrics tab" /%} ## Data collected{% #data-collected %} The following tables list the OpenTelemetry runtime metrics used in Datadog's out-of-the-box in-app experiences. Each row maps an OpenTelemetry metric to the equivalent Datadog metric name, along with a description and any attribute conditions Datadog uses to identify the correct data point: - **Transform**: shown when both the OpenTelemetry metric and the Datadog metric require attribute filters to uniquely identify the data point. - **Filter**: shown when only the OpenTelemetry metric requires an attribute filter. For Collector processor configuration required to make these metrics compatible with Datadog, see the Collector configuration instructions above. {% tab title="Java" %} ### JVM Instrumentation These metrics are collected when using the latest version of the OpenTelemetry Java SDK. | OTEL | DATADOG | DESCRIPTION | FILTER | | -------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | jvm.buffer.count | jvm.buffer_pool.direct.count | Number of buffers in the pool. | `jvm.buffer.pool.name`: `direct` | | jvm.buffer.count | jvm.buffer_pool.mapped.count | Number of buffers in the pool. | `jvm.buffer.pool.name`: `mapped` | | jvm.buffer.memory.limit | jvm.buffer_pool.direct.capacity | Measure of total memory capacity of buffers. | `jvm.buffer.pool.name`: `direct` | | jvm.buffer.memory.limit | jvm.buffer_pool.mapped.capacity | Measure of total memory capacity of buffers. | `jvm.buffer.pool.name`: `mapped` | | jvm.buffer.memory.used | jvm.buffer_pool.direct.used | Measure of memory used by buffers. | `jvm.buffer.pool.name`: `direct` | | jvm.buffer.memory.used | jvm.buffer_pool.mapped.used | Measure of memory used by buffers. | `jvm.buffer.pool.name`: `mapped` | | jvm.class.count | jvm.loaded_classes | Number of classes currently loaded. | | jvm.cpu.recent_utilization | jvm.cpu_load.process | Recent CPU utilization for the process as reported by the JVM. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike `system.cpu.utilization`). [Reference](https://docs.oracle.com/en/java/javase/17/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad%28%29). | | jvm.file_descriptor.count | jvm.os.open_file_descriptors | Number of open file descriptors as reported by the JVM. | | jvm.memory.used | jvm.gc.old_gen_size | Measure of memory used. | `jvm.memory.pool.name`: `g1_old_gen, zgc_old_generation, tenured_gen, ps_old_gen, cms_old_gen``jvm.memory.type`: `heap` | | jvm.memory.used | jvm.gc.eden_size | Measure of memory used. | `jvm.memory.pool.name`: `g1_eden_space, zgc_young_generation, eden_space, ps_eden_space, par_eden_space``jvm.memory.type`: `heap` | | jvm.memory.used | jvm.gc.survivor_size | Measure of memory used. | `jvm.memory.pool.name`: `g1_survivor_space, survivor_space, ps_survivor_space, par_survivor_space``jvm.memory.type`: `heap` | | jvm.memory.used | jvm.gc.metaspace_size | Measure of memory used. | `jvm.memory.pool.name`: `metaspace``jvm.memory.type`: `non_heap` | | jvm.system.cpu.utilization | jvm.cpu_load.system | Recent CPU utilization for the whole system as reported by the JVM. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike `system.cpu.utilization`). [Reference](https://docs.oracle.com/en/java/javase/17/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html#getCpuLoad%28%29). | ### JVM Contrib These metrics are collected when using the OpenTelemetry JMX Metric Scraper. | OTEL | DATADOG | DESCRIPTION | FILTER | | ---------------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | jvm.classes.loaded | jvm.loaded_classes | The number of loaded classes | | jvm.gc.collections.count | jvm.gc.minor_collection_count{type: GarbageCollector} | The total number of garbage collections that have occurred | `name`: `copy, ps_scavenge, parnew, g1_young_generation, zgc_minor_cycles, zgc_minor_pauses` | | jvm.gc.collections.count | jvm.gc.major_collection_count{type: GarbageCollector} | The total number of garbage collections that have occurred | `name`: `marksweepcompact, ps_marksweep, concurrentmarksweep, g1_mixed_generation, g1_old_generation, shenandoah_cycles, zgc_major_cycles, zgc_major_pauses` | | jvm.gc.collections.elapsed | jvm.gc.minor_collection_time{type: GarbageCollector} | The approximate accumulated collection elapsed time | `name`: `copy, ps_scavenge, parnew, g1_young_generation, zgc_minor_cycles, zgc_minor_pauses` | | jvm.gc.collections.elapsed | jvm.gc.major_collection_time{type: GarbageCollector} | The approximate accumulated collection elapsed time | `name`: `marksweepcompact, ps_marksweep, concurrentmarksweep, g1_mixed_generation, g1_old_generation, shenandoah_cycles, zgc_major_cycles, zgc_major_pauses` | | jvm.memory.heap.committed | jvm.heap_memory_committed | The amount of memory that is guaranteed to be available for the heap | | jvm.memory.heap.init | jvm.heap_memory_init | The initial amount of memory that the JVM requests from the operating system for the heap | | jvm.memory.heap.max | jvm.heap_memory_max | The maximum amount of memory can be used for the heap | | jvm.memory.heap.used | jvm.heap_memory | The current heap memory usage | | jvm.memory.nonheap.committed | jvm.non_heap_memory_committed | The amount of memory that is guaranteed to be available for non-heap purposes | | jvm.memory.nonheap.init | jvm.non_heap_memory_init | The initial amount of memory that the JVM requests from the operating system for non-heap purposes | | jvm.memory.nonheap.max | jvm.non_heap_memory_max | The maximum amount of memory can be used for non-heap purposes | | jvm.memory.nonheap.used | jvm.non_heap_memory | The current non-heap memory usage | | jvm.memory.pool.used | jvm.gc.metaspace_size | The current memory pool memory usage | `name`: `metaspace` | | jvm.memory.pool.used | jvm.gc.old_gen_size | The current memory pool memory usage | `name`: `g1_old_gen, zgc_old_generation, tenured_gen, ps_old_gen, cms_old_gen` | | jvm.memory.pool.used | jvm.gc.eden_size | The current memory pool memory usage | `name`: `g1_eden_space, zgc_young_generation, eden_space, ps_eden_space, par_eden_space` | | jvm.memory.pool.used | jvm.gc.survivor_size | The current memory pool memory usage | `name`: `g1_survivor_space, survivor_space, ps_survivor_space, par_survivor_space` | | jvm.threads.count | jvm.thread_count | The current number of threads | ### Deprecated JVM Metrics These metrics are collected when using OpenTelemetry Java SDK versions 1.32.0 and earlier. | OTEL | DATADOG | DESCRIPTION | FILTER | | ------------------------------------------ | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | process.runtime.jvm.buffer.count | jvm.buffer_pool.direct.count | Number of buffers in the pool. | `pool`: `direct` | | process.runtime.jvm.buffer.count | jvm.buffer_pool.mapped.count | Number of buffers in the pool. | `pool`: `mapped` | | process.runtime.jvm.buffer.usage | jvm.buffer_pool.direct.used | Measure of memory used by buffers. | `pool`: `direct` | | process.runtime.jvm.buffer.usage | jvm.buffer_pool.mapped.used | Measure of memory used by buffers. | `pool`: `mapped` | | process.runtime.jvm.classes.current_loaded | jvm.loaded_classes | Number of classes currently loaded. | | process.runtime.jvm.cpu.utilization | jvm.cpu_load.process | Recent CPU utilization for the process. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike `system.cpu.utilization`). [Reference](https://docs.oracle.com/en/java/javase/17/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad%28%29). | | process.runtime.jvm.memory.usage | jvm.gc.old_gen_size | Measure of memory used. | `pool`: `g1_old_gen, zgc_old_generation, tenured_gen, ps_old_gen, cms_old_gen``type`: `heap` | | process.runtime.jvm.memory.usage | jvm.gc.eden_size | Measure of memory used. | `pool`: `g1_eden_space, zgc_young_generation, eden_space, ps_eden_space, par_eden_space``type`: `heap` | | process.runtime.jvm.memory.usage | jvm.gc.survivor_size | Measure of memory used. | `pool`: `g1_survivor_space, survivor_space, ps_survivor_space, par_survivor_space``type`: `heap` | | process.runtime.jvm.memory.usage | jvm.gc.metaspace_size | Measure of memory used. | `pool`: `metaspace``type`: `non_heap` | | process.runtime.jvm.system.cpu.utilization | jvm.cpu_load.system | Recent CPU utilization for the whole system as reported by the JVM. Note: The value range is [0.0,1.0]. This utilization is not defined as being for the specific interval since last measurement (unlike `system.cpu.utilization`). [Reference](https://docs.oracle.com/en/java/javase/17/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html#getCpuLoad%28%29). | {% /tab %} {% tab title="Go" %} ### Go Runtime Metrics These metrics are collected by the OpenTelemetry Go [runtime instrumentation package](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime). | OTEL | DATADOG | DESCRIPTION | | ------------------------------------ | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | go.config.gogc | runtime.go.metrics.gc_gogc.percent | Heap size target percentage configured by the user, otherwise 100. Note: The value range is [0.0,100.0]. Computed from `/gc/gogc:percent`. | | go.goroutine.count | runtime.go.metrics.sched_goroutines.goroutines | Count of live goroutines. Note: Computed from `/sched/goroutines:goroutines`. | | go.memory.allocated | runtime.go.metrics.gc_heap_allocs.bytes | Memory allocated to the heap by the application. Note: Computed from `/gc/heap/allocs:bytes`. | | go.memory.allocations | runtime.go.metrics.gc_heap_allocs.objects | Count of allocations to the heap by the application. Note: Computed from `/gc/heap/allocs:objects`. | | go.memory.gc.goal | runtime.go.metrics.gc_heap_goal.bytes | Heap size target for the end of the GC cycle. Note: Computed from `/gc/heap/goal:bytes`. | | go.memory.limit | runtime.go.metrics.gc_gomemlimit.bytes | Go runtime memory limit configured by the user, if a limit exists. Note: Computed from `/gc/gomemlimit:bytes`. This metric is excluded if the limit obtained from the Go runtime is math.MaxInt64. | | go.processor.limit | runtime.go.metrics.sched_gomaxprocs.threads | The number of OS threads that can execute user-level Go code simultaneously. Note: Computed from `/sched/gomaxprocs:threads`. | | process.runtime.go.cgo.calls | runtime.go.num_cgo_call | Number of cgo calls made by the current process | | process.runtime.go.gc.count | runtime.go.mem_stats.num_gc | Number of completed garbage collection cycles | | process.runtime.go.gc.pause_total_ns | runtime.go.mem_stats.pause_total_ns | Cumulative nanoseconds in GC stop-the-world pauses since the program started | | process.runtime.go.goroutines | runtime.go.num_goroutine | Number of goroutines that currently exist | | process.runtime.go.mem.heap_alloc | runtime.go.mem_stats.heap_alloc | Bytes of allocated heap objects | | process.runtime.go.mem.heap_idle | runtime.go.mem_stats.heap_idle | Bytes in idle (unused) spans | | process.runtime.go.mem.heap_inuse | runtime.go.mem_stats.heap_inuse | Bytes in in-use spans | | process.runtime.go.mem.heap_objects | runtime.go.mem_stats.heap_objects | Number of allocated heap objects | | process.runtime.go.mem.heap_released | runtime.go.mem_stats.heap_released | Bytes of idle spans whose physical memory has been returned to the OS | | process.runtime.go.mem.heap_sys | runtime.go.mem_stats.heap_sys | Bytes of heap memory obtained from the OS | {% /tab %} {% tab title=".NET" %} ### .NET System.Runtime These metrics are emitted by the .NET runtime's built-in `System.Runtime` meter on .NET 9.0 and later. The OpenTelemetry SDK collects and exports them automatically. | OTEL | DATADOG | DESCRIPTION | FILTER | | ----------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | dotnet.gc.last_collection.heap.size | runtime.dotnet.gc.size.gen0 | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: `System.Runtime`; Added in: .NET 9.0. This metric reports the same values as calling [`GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes`](https://learn.microsoft.com/dotnet/api/system.gcgenerationinfo.sizeafterbytes). | `gc.heap.generation`: `gen0` | | dotnet.gc.last_collection.heap.size | runtime.dotnet.gc.size.gen1 | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: `System.Runtime`; Added in: .NET 9.0. This metric reports the same values as calling [`GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes`](https://learn.microsoft.com/dotnet/api/system.gcgenerationinfo.sizeafterbytes). | `gc.heap.generation`: `gen1` | | dotnet.gc.last_collection.heap.size | runtime.dotnet.gc.size.gen2 | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: `System.Runtime`; Added in: .NET 9.0. This metric reports the same values as calling [`GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes`](https://learn.microsoft.com/dotnet/api/system.gcgenerationinfo.sizeafterbytes). | `gc.heap.generation`: `gen2` | | dotnet.gc.last_collection.heap.size | runtime.dotnet.gc.size.loh | The managed GC heap size (including fragmentation), as observed during the latest garbage collection. Note: Meter name: `System.Runtime`; Added in: .NET 9.0. This metric reports the same values as calling [`GC.GetGCMemoryInfo().GenerationInfo.SizeAfterBytes`](https://learn.microsoft.com/dotnet/api/system.gcgenerationinfo.sizeafterbytes). | `gc.heap.generation`: `loh` | | dotnet.process.cpu.time | runtime.dotnet.cpu.user | CPU time used by the process. Note: Meter name: `System.Runtime`; Added in: .NET 9.0. This metric reports the same values as accessing the corresponding processor time properties on [`System.Diagnostics.Process`](https://learn.microsoft.com/dotnet/api/system.diagnostics.process). | `cpu.mode`: `user` | | dotnet.process.cpu.time | runtime.dotnet.cpu.system | CPU time used by the process. Note: Meter name: `System.Runtime`; Added in: .NET 9.0. This metric reports the same values as accessing the corresponding processor time properties on [`System.Diagnostics.Process`](https://learn.microsoft.com/dotnet/api/system.diagnostics.process). | `cpu.mode`: `system` | ### .NET Contrib Runtime These metrics are collected by the [OpenTelemetry.Instrumentation.Runtime](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Runtime) package. On .NET 9.0 and later, these overlap with the System.Runtime metrics above. | OTEL | DATADOG | DESCRIPTION | FILTER | | ------------------------------------------------ | ------------------------------------ | --------------------------------------------------------------- | -------------------- | | process.runtime.dotnet.gc.heap.size | runtime.dotnet.gc.size.gen0 | The heap size (including fragmentation), as observed during the | `generation`: `gen0` | | process.runtime.dotnet.gc.heap.size | runtime.dotnet.gc.size.gen1 | The heap size (including fragmentation), as observed during the | `generation`: `gen1` | | process.runtime.dotnet.gc.heap.size | runtime.dotnet.gc.size.gen2 | The heap size (including fragmentation), as observed during the | `generation`: `gen2` | | process.runtime.dotnet.gc.heap.size | runtime.dotnet.gc.size.loh | The heap size (including fragmentation), as observed during the | `generation`: `loh` | | process.runtime.dotnet.thread_pool.threads.count | runtime.dotnet.threads.workers_count | The number of thread pool threads that currently exist. | ### .NET Contrib Process These metrics are collected by the [OpenTelemetry.Instrumentation.Process](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Process) package. | OTEL | DATADOG | DESCRIPTION | | -------------------- | ---------------------------- | ---------------------- | | process.thread.count | runtime.dotnet.threads.count | Process threads count. | {% /tab %} {% tab title="Node.js" %} ### Node.js Contrib Runtime These metrics are emitted from auto-instrumentation with the latest version of the OpenTelemetry Node.js SDK. | OTEL | DATADOG | DESCRIPTION | TRANSFORM | | ------------------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | nodejs.eventloop.utilization | runtime.node.event_loop.utilization | Event loop utilization. Note: The value range is [0.0, 1.0] and can be retrieved from [`performance.eventLoopUtilization([utilization1[, utilization2]])`](https://nodejs.org/api/perf_hooks.html#performanceeventlooputilizationutilization1-utilization2) | | v8js.memory.heap.limit | runtime.node.heap.size.by.space | Total heap memory size pre-allocated. Note: The value can be retrieved from value `space_size` of [`v8.getHeapSpaceStatistics()`](https://nodejs.org/api/v8.html#v8getheapspacestatistics) | Rename attribute key `heap_space` to `v8js.heap.space.name` | | v8js.memory.heap.space.available_size | runtime.node.heap.available_size.by.space | Heap space available size. Note: Value can be retrieved from value `space_available_size` of [`v8.getHeapSpaceStatistics()`](https://nodejs.org/api/v8.html#v8getheapspacestatistics) | Rename attribute key `heap_space` to `v8js.heap.space.name` | | v8js.memory.heap.space.physical_size | runtime.node.heap.physical_size.by.space | Committed size of a heap space. Note: Value can be retrieved from value `physical_space_size` of [`v8.getHeapSpaceStatistics()`](https://nodejs.org/api/v8.html#v8getheapspacestatistics) | Rename attribute key `heap_space` to `v8js.heap.space.name` | | v8js.memory.heap.used | runtime.node.heap.used_size.by.space | Heap Memory size allocated. Note: The value can be retrieved from value `space_used_size` of [`v8.getHeapSpaceStatistics()`](https://nodejs.org/api/v8.html#v8getheapspacestatistics) | Rename attribute key `heap_space` to `v8js.heap.space.name` | ### Node.js Contrib Host These metrics are collected by the [@opentelemetry/host-metrics](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/host-metrics) package. This package is not included in OpenTelemetry automatic instrumentation and must be installed and configured manually. | OTEL | DATADOG | DESCRIPTION | FILTER | | -------------------- | --------------------- | ------------------------------------- | --------------------------------- | | process.memory.usage | runtime.node.mem.rss | The amount of physical memory in use. | | process.memory.usage | runtime.node.mem.rss | The amount of physical memory in use. | | system.memory.usage | system.mem.total | Bytes of memory in use. | | system.memory.usage | system.mem.usable | Bytes of memory in use. | `state`: `free, cached, buffered` | | system.memory.usage | system.mem.free | Bytes of memory in use. | `state`: `free` | | system.memory.usage | runtime.node.mem.free | Bytes of memory in use. | `system.memory.state`: `free` | | system.memory.usage | runtime.node.mem.free | Bytes of memory in use. | `system.memory.state`: `free` | {% /tab %} {% tab title="Python" %} ### Python Runtime Metrics These metrics are collected by the [`opentelemetry-instrumentation-system-metrics`](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-system-metrics) package. The following table lists the conceptual equivalences between OpenTelemetry and Datadog Python runtime metrics. There are no direct metric-name mappings because the metric types differ between the two systems. | OTEL | Datadog | Description | Filter | | -------------------------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------- | --------------------- | | `process.cpu.time` | `runtime.python.cpu.time.sys` | Number of seconds executing in the kernel. | `type`: `system` | | `process.cpu.time` | `runtime.python.cpu.time.user` | Number of seconds executing outside the kernel. | `type`: `user` | | `process.cpu.utilization` | `runtime.python.cpu.percent` | CPU utilization percentage. OTel divides the raw value by 100 times the number of CPU cores. | | `process.context_switches` | `runtime.python.cpu.ctx_switch.voluntary` | Number of voluntary context switches. | `type`: `voluntary` | | `process.context_switches` | `runtime.python.cpu.ctx_switch.involuntary` | Number of involuntary context switches. | `type`: `involuntary` | | `process.runtime.{python_implementation}.gc_count` | `runtime.python.gc.count.gen0` | Number of generation 0 objects. | `count`: `0` | | `process.runtime.{python_implementation}.gc_count` | `runtime.python.gc.count.gen1` | Number of generation 1 objects. | `count`: `1` | | `process.runtime.{python_implementation}.gc_count` | `runtime.python.gc.count.gen2` | Number of generation 2 objects. | `count`: `2` | | `process.memory.usage` | `runtime.python.mem.rss` | Resident set memory. | | `process.thread.count` | `runtime.python.thread_count` | Number of threads. | {% /tab %} ## Troubleshooting{% #troubleshooting %} ### Metric name mapping{% #metric-name-mapping %} OpenTelemetry runtime metrics are mapped to Datadog by metric name. Do not rename host metrics for OpenTelemetry runtime metrics as this breaks the mapping. ## Further reading{% #further-reading %} - [APM Runtime Metrics](https://docs.datadoghq.com/tracing/metrics/runtime_metrics.md) - [OpenTelemetry Metrics Mapping](https://docs.datadoghq.com/opentelemetry/mapping/metrics_mapping.md)