Migrate to OpenTelemetry Collector version 0.120.0+
OTel Collector Contrib version 0.120.0 introduced breaking changes to metric names as part of the upgrade to Prometheus 3.0. After upgrading to this version of the OpenTelemetry Collector, you may notice differences in metric values displayed in Datadog.
These breaking changes are not introduced by or directly related to Datadog. They impact all OpenTelemetry users who use Prometheus. For a complete list of changes, see the
update to the Prometheus receiver in the Collector or the Prometheus 3.0
migration guide.
Changes to Collector internal metric names
Collector Internal Metrics sent using the latest Collector version have the following changes:
- Dots (
.) in internal collector metrics and resource attributes scraped by Prometheus are no longer replaced with underscores (_) by default. - The
otelcol_ prefix is no longer added to metric names.
For example:
| Before 0.120.0 | After 0.120.0 |
|---|
otelcol_datadog_trace_agent_otlp_traces | datadog.trace_agent.otlp.traces |
otelcol_datadog_trace_agent_otlp_spans | datadog.trace_agent.otlp.spans |
otelcol_datadog_trace_agent_otlp_payload | datadog.trace_agent.otlp.payload |
otelcol_datadog_trace_agent_trace_writer_events | datadog.trace_agent.trace_writer.events |
As a result, Datadog has updated two out-of-the-box dashboards affected by this upgrade:
OpenTelemetry Collector health dashboard
Queries on the OpenTelemetry Collector health dashboard were modified to be compatible with metric names sent from both older (< 0.120.0) and newer (0.120.0+) versions of the Collector.
If you are using a cloned version of this dashboard or have monitors that query metric names from older Collector versions, you may need to manually update them using the equiv_otel() function.
APM Datadog Trace Agent dashboard
Queries on the APM Datadog Trace Agent dashboard were updated with filters to exclude sources datadogexporter and datadogconnector to prevent metric collisions with OpenTelemetry sources that emit the same metric names. This dashboard is designed to show only Trace Agent data, and the update ensures that data from these sources doesn’t mix with OpenTelemetry data.
Only the out-of-the-box dashboard template was updated. If you are using a cloned version of this dashboard, you may need to manually update queries on custom dashboards to exclude sources datadogexporter and datadogconnector using:
source NOT IN (datadogexporter, datadogconnector)
Changes to Prometheus Server reader defaults
If you use default configurations for your OpenTelemetry Collector's telemetry settings, you will not be impacted by these changes.
You are only impacted if you have explicitly configured the Prometheus reader with custom settings, such as:
service:
telemetry:
metrics:
level: detailed
readers:
- pull:
exporter:
prometheus:
host: localhost
port: 8888
If you are affected by these changes, you may see differences in metric names, such as suffix changes and unit additions.
To revert to the previous behavior, add these three parameters to your existing Prometheus reader configuration:
without_scope_info: true
without_type_suffix: true
without_units: true
For questions or assistance, contact Datadog support.
Further reading
Additional helpful documentation, links, and articles:
