OTel Collector Contrib version 0.120.0 introduced breaking changes to metric names as part of the upgrade to Prometheus 3.0. After upgrading to this version of the OpenTelemetry Collector, you may notice differences in metric values displayed in Datadog.

Changes to Collector internal metric names

Collector Internal Metrics sent using the latest Collector version have the following changes:

• Dots (.) in internal collector metrics and resource attributes scraped by Prometheus are no longer replaced with underscores (_) by default.
• The otelcol_ prefix is no longer added to metric names.

For example:

As a result, Datadog has updated two out-of-the-box dashboards affected by this upgrade:

• OpenTelemetry Collector Health Dashboard
• APM Datadog Trace Agent Dashboard

OpenTelemetry Collector health dashboard

Queries on the OpenTelemetry Collector health dashboard were modified to be compatible with metric names sent from both older (< 0.120.0) and newer (0.120.0+) versions of the Collector.

If you are using a cloned version of this dashboard or have monitors that query metric names from older Collector versions, you may need to manually update them using the equiv_otel() function.

APM Datadog Trace Agent dashboard

Queries on the APM Datadog Trace Agent dashboard were updated with filters to exclude sources datadogexporter and datadogconnector to prevent metric collisions with OpenTelemetry sources that emit the same metric names. This dashboard is designed to show only Trace Agent data, and the update ensures that data from these sources doesn’t mix with OpenTelemetry data.

Only the out-of-the-box dashboard template was updated. If you are using a cloned version of this dashboard, you may need to manually update queries on custom dashboards to exclude sources datadogexporter and datadogconnector using:

source NOT IN (datadogexporter, datadogconnector)

Changes to Prometheus Server reader defaults

You are only impacted if you have explicitly configured the Prometheus reader with custom settings, such as:

service:
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: localhost
                port: 8888

If you are affected by these changes, you may see differences in metric names, such as suffix changes and unit additions.

To revert to the previous behavior, add these three parameters to your existing Prometheus reader configuration:

without_scope_info: true
without_type_suffix: true
without_units: true

For questions or assistance, contact Datadog support.

Further reading

Additional helpful documentation, links, and articles:

Migrate to OpenTelemetry Collector version 0.95.0+DOCUMENTATION OpenTelemetry Collector Health DashboardDOCUMENTATION APM Datadog Trace Agent DashboardDOCUMENTATION Prometheus 3.0 Migration GuideEXTERNAL SITE