Tracing Application Metrics
Overview
Tracing application metrics are collected after enabling trace collection and instrumenting your application. These metrics are available for dashboards and monitors. The trace metrics namespace is formatted as:
trace.<SPAN_NAME>.<METRIC_SUFFIX>
trace.<SPAN_NAME>.<METRIC_SUFFIX>.<2ND_PRIM_TAG>_service
With the following definitions:
<SPAN_NAME>
- The name of the operation or
span.name
(examples: redis.command
, pylons.request
, rails.request
, mysql.query
). <METRIC_SUFFIX>
- The name of the metric (examples:
duration
, hits
, span_count
). See the section below. <2ND_PRIM_TAG>
- If the metric name accounts for the second primary tag, this tag is part of the metric name.
<TAGS>
- Trace metrics tags, possible tags are:
env
, service
, version
, resource
, sublayer_type
, sublayer_service
, http.status_code
, http.status_class
, Datadog Agent tags (including the host and second primary tag). Note: Tags set on spans do not count and will not be available as tags for your traces metrics.
Metric suffix
Hits
trace.<SPAN_NAME>.hits
- Prerequisite: This metric exists for any APM service.
Description: Represent the count of hits for a given span.
Metric type: COUNT.
Tags: env
, service
, version
, resource
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag. trace.<SPAN_NAME>.hits.by_http_status
- Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Represent the count of hits for a given span break down by HTTP status code.
Metric type: COUNT.
Tags: env
, service
, version
, resource
, http.status_class
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag.
Latency distribution
trace.<SPAN_NAME>
- Prerequisite: This metric exists for any APM service.
Description: Represent the latency distribution for all services, resources, and versions across different environments and second primary tags.
Metric type: DISTRIBUTION.
Tags: env
, service
, resource
, resource_name
, version
, synthetics
, and the second primary tag.
Errors
trace.<SPAN_NAME>.errors
- Prerequisite: This metric exists for any APM service.
Description: Represent the count of errors for a given span.
Metric type: COUNT.
Tags: env
, service
, version
, resource
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag. trace.<SPAN_NAME>.errors.by_http_status
- Prerequisite: This metric exists for any APM service.
Description: Represent the count of errors for a given span.
Metric type: COUNT.
Tags: env
, service
, version
, resource
, http.status_class
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag.
Span count
Note: This is a deprecated namespace.
trace.<SPAN_NAME>.span_count
- Prerequisite: This metric exists for any APM service.
Description: Represent the amount of spans collected on a given interval.
Metric type: COUNT.
Tags: env
, service
, resource
, all host tags on from the Datadog Host Agent, and the second primary tag. trace.<SPAN_NAME>.span_count.by_http_status
- Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Represent the amount of spans collected on a given interval break down by HTTP status.
Metric type: COUNT.
Tags: env
, service
, resource
, http.status_class
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag.
Duration
trace.<SPAN_NAME>.duration
- Prerequisite: This metric exists for any APM service.
Description: [LEGACY] Measure the total time for a collection of spans within a time interval, including child spans seen in the collecting service. This metric used to generate the “% exec time for downstream services” graph. When trace.<SPAN_NAME>.duration
is divided by trace.<SPAN_NAME>.hits
, the result can yield an average latency, but this is not the recommended approach for calculating the average latency. Instead, refer to the Latency Distribution section for average latency calculations.
Metric type: GAUGE.
Tags: env
, service
, resource
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag.
Duration by
trace.<SPAN_NAME>.duration.by_http_status
- Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Measure the total time for a collection of spans for each HTTP status. Specifically, it is the relative share of time spent by all spans over an interval and a given HTTP status - including time spent waiting on child processes.
Metric type: GAUGE.
Tags: env
, service
, resource
, http.status_class
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag. trace.<SPAN_NAME>.duration.by_service
- Prerequisite: This metric exists for any APM service.
Description: Measure the total time spent actually processing for each service (as in, it excludes time spent waiting on child processes).
Metric type: GAUGE.
Tags: env
, service
, resource
, sublayer_service
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag. trace.<SPAN_NAME>.duration.by_type
- Prerequisite: This metric exists for any APM service.
Description: Measure the total time spent actually processing for each Service type.
Metric type: GAUGE.
Tags: env
, service
, resource
, sublayer_type
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag. trace.<SPAN_NAME>.duration.by_type.by_http_status
- Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Measure the total time spent actually processing for each Service type and HTTP status.
Metric type: GAUGE.
Tags: env
, service
, resource
, sublayer_type
, http.status_class
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag. trace.<SPAN_NAME>.duration.by_service.by_http_status
- Prerequisite: This metric exists for HTTP/WEB APM services if http metadata exists.
Description: Measure the total time spent actually processing for each Service and HTTP status.
Metric type: GAUGE.
Tags: env
, service
, resource
, sublayer_service
, http.status_class
, http.status_code
, all host tags from the Datadog Host Agent, and the second primary tag.
Apdex
trace.<SPAN_NAME>.apdex
- Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each web service.
Metric type: GAUGE.
Tags: env
, service
, resource
/ resource_name
, version
, synthetics
, and the second primary tag.
The following legacy apdex metrics are deprecated.
trace.<SPAN_NAME>.apdex.by.resource_<2ND_PRIM_TAG>_service
- Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Represents the Apdex score for all combination of resources, 2nd primary tags and services.
Metric type: GAUGE.
Tags: env
, service
, resource
/ resource_name
, and the second primary tag. trace.<SPAN_NAME>.apdex.by.resource_service
- Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each combination of resources and web services.
Metric type: GAUGE.
Tags: env
, service
, and resource
/ resource_name
. trace.<SPAN_NAME>.apdex.by.<2ND_PRIM_TAG>_service
- Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each combination of 2nd primary tag and web services.
Metric type: GAUGE.
Tags: env
, service
, and the second primary tag. trace.<SPAN_NAME>.apdex.by.service
- Prerequisite: This metric exists for any HTTP/WEB APM service.
Description: Measures the Apdex score for each web service.
Metric type: GAUGE.
Tags: env
and service
.
Further Reading
Additional helpful documentation, links, and articles: