Tracing Application Metrics

Tracing Application Metrics

Overview

Tracing application metrics are collected after enabling trace collection and instrumenting your application. These metrics are available for dashboards and monitors. The trace metrics namespace is formatted as:

  • trace.<SPAN_NAME>.<METRIC_SUFFIX>
  • trace.<SPAN_NAME>.<METRIC_SUFFIX>.<2ND_PRIM_TAG>_service

With the following definitions:

<SPAN_NAME>
The name of the operation or span.name (examples: redis.command, pylons.request, rails.request, mysql.query).
<METRIC_SUFFIX>
The name of the metric (examples: duration, hits, span_count). See the section below.
<2ND_PRIM_TAG>
If the metric name accounts for the second primary tag, this tag is part of the metric name.
<TAGS>
Trace metrics tags, possible tags are: env, service, version, resource, sublayer_type, sublayer_service, http.status_code, http.status_class, Datadog Agent tags (including the host and second primary tag). Note: Tags set on spans do not count and will not be available as tags for your traces metrics.

Metric suffix

Hits

trace.<SPAN_NAME>.hits
Prerequisite: This metric exists for any APM service.
Description: Represent the count of hits for a given span.
Metric type: COUNT.
Tags: env, service, version, resource, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.
trace.<SPAN_NAME>.hits.by_http_status
Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Represent the count of hits for a given span break down by HTTP status code.
Metric type: COUNT.
Tags: env, service, version, resource, http.status_class, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.

Latency distribution

trace.<SPAN_NAME>
Prerequisite: This metric exists for any APM service.
Description: Represent the latency distribution for all services, resources, and versions across different environments and second primary tags.
Metric type: DISTRIBUTION.
Tags: env, service, resource, resource_name, version, synthetics, and the second primary tag.

Errors

trace.<SPAN_NAME>.errors
Prerequisite: This metric exists for any APM service.
Description: Represent the count of errors for a given span.
Metric type: COUNT.
Tags: env, service, version, resource, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.
trace.<SPAN_NAME>.errors.by_http_status
Prerequisite: This metric exists for any APM service.
Description: Represent the count of errors for a given span.
Metric type: COUNT.
Tags: env, service, version, resource, http.status_class, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.

Span count

Note: This is a deprecated namespace.

trace.<SPAN_NAME>.span_count
Prerequisite: This metric exists for any APM service.
Description: Represent the amount of spans collected on a given interval.
Metric type: COUNT.
Tags: env, service, resource, all host tags on from the Datadog Host Agent, and the second primary tag.
trace.<SPAN_NAME>.span_count.by_http_status
Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Represent the amount of spans collected on a given interval break down by HTTP status.
Metric type: COUNT.
Tags: env, service, resource, http.status_class, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.

Duration

trace.<SPAN_NAME>.duration
Prerequisite: This metric exists for any APM service.
Description: Measure the total time for a collection of spans. Specifically, it is the total time spent by all spans over an interval - including time spent waiting on child processes.
Metric type: GAUGE.
Tags: env, service, resource, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.

Duration by

trace.<SPAN_NAME>.duration.by_http_status
Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Measure the total time for a collection of spans for each HTTP status. Specifically, it is the relative share of time spent by all spans over an interval and a given HTTP status - including time spent waiting on child processes.
Metric type: GAUGE.
Tags: env, service, resource, http.status_class, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.
trace.<SPAN_NAME>.duration.by_service
Prerequisite: This metric exists for any APM service.
Description: Measure the total time spent actually processing for each service (i.e. it excludes time spent waiting on child processes).
Metric type: GAUGE.
Tags: env, service, resource, sublayer_service, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.
trace.<SPAN_NAME>.duration.by_type
Prerequisite: This metric exists for any APM service.
Description: Measure the total time spent actually processing for each Service type.
Metric type: GAUGE.
Tags: env, service, resource, sublayer_type, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.
trace.<SPAN_NAME>.duration.by_type.by_http_status
Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Measure the total time spent actually processing for each Service type and HTTP status.
Metric type: GAUGE.
Tags: env, service, resource, sublayer_type, http.status_class, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.
trace.<SPAN_NAME>.duration.by_service.by_http_status
Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Measure the total time spent actually processing for each Service and HTTP status.
Metric type: GAUGE.
Tags: env, service, resource, sublayer_service, http.status_class, http.status_code, all host tags from the Datadog Host Agent, and the second primary tag.

Apdex

trace.<SPAN_NAME>.apdex
Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each web service.
Metric type: GAUGE.
Tags: env, service, resource / resource_name, version, synthetics, and the second primary tag.

The following legacy apdex metrics are deprecated.

trace.<SPAN_NAME>.apdex.by.resource_<2ND_PRIM_TAG>_service
Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Represents the Apdex score for all combination of resources, 2nd primary tags and services.
Metric type: GAUGE.
Tags: env, service, resource / resource_name, and the second primary tag.
trace.<SPAN_NAME>.apdex.by.resource_service
Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each combination of resources and web services.
Metric type: GAUGE.
Tags: env, service, and resource / resource_name.
trace.<SPAN_NAME>.apdex.by.<2ND_PRIM_TAG>_service
Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each combination of 2nd primary tag and web services.
Metric type: GAUGE.
Tags: env, service, and the second primary tag.
trace.<SPAN_NAME>.apdex.by.service
Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each web service.
Metric type: GAUGE.
Tags: env and service.

Further Reading