Tracing Application Metrics
Datadog's Research Report: The State of Serverless Report: The State of Serverless
home Home docs Docs api API

Tracing Application Metrics

Overview

The trace metrics namespace is trace.<SPAN_NAME>.<METRICS_suffix>{<TAGS>}. Tracing application metrics are collected after enabling trace collection and instrumenting your application. These metrics are available for dashboards and monitors.

Trace metric names are formatted as trace.<SPAN_NAME>.<METRIC_SUFFIX>{<TAGS>} or trace.<SPAN_NAME>.<2ND_PRIM_TAG>{<TAGS>}.

ParameterDescription
<SPAN_NAME>The name of the operation or span.name (examples: redis.command, pylons.request, rails.request, mysql.query).
<METRIC_SUFFIX>The name of the metric (examples: duration, hits, span_count).
<2ND_PRIM_TAG>If the metric name accounts for the second primary tag, this tag is part of the metric name.
<TAGS>Trace metrics tags, possible tags are: env, service, resource, sublayer_type, sublayer_service, http.status_code, http.status_class, Datadog Agent tags (including the host and second primary tag).

Metric Suffix

The following metrics have all the second primary tag and Datadog Agent tags attached to it.

Duration

  • trace.<SPAN_NAME>.duration:

    • Pre-requisit: This metric exists for any APM service.
    • Description: Measure the total time for a collection of spans. Specifically, it is the total time spent by all spans over an interval - including time spent waiting on child processes.
    • Metric type: GAUGE
    • Tags: env, service, resource, and http.status_code.

  • trace.<SPAN_NAME>.duration.by_http_status:

    • Pre-requisit: This metric exists for HTTP/WEB APM services if http metadata exists.
    • Description: Measure the total time for a collection of spans for each HTTP status. Specifically, it is the relative share of time spent by all spans over an interval and a given HTTP status - including time spent waiting on child processes.
    • Metric type: GAUGE
    • Tags: env, service, resource, http.status_class and http.status_code.

  • trace.<SPAN_NAME>.duration.by_service:

    • Pre-requisit: This metric exists for any APM service.
    • Description: Measures the total time spent actually processing for each service (i.e. it excludes time spent waiting on child processes).
    • Metric type: GAUGE
    • Tags: env, service, resource, sublayer_service, and http.status_code.

  • trace.<SPAN_NAME>.duration.by_type:

    • Pre-requisit: This metric exists for any APM service.
    • Description: Measures the total time spent actually processing for each Service type.
    • Metric type: GAUGE
    • Tags: env, service, resource, sublayer_type, and http.status_code.

  • trace.<SPAN_NAME>.duration.by_type.by_http_status:

    • Pre-requisit: This metric exists for HTTP/WEB APM services if http metadata exists.
    • Description: Measures the total time spent actually processing for each Service type and HTTP status.
    • Metric type: GAUGE
    • Tags: env, service, resource, sublayer_type, http.status_class, and http.status_code.

  • trace.<SPAN_NAME>.duration.by_service.by_http_status:

    • Pre-requisit: This metric exists for HTTP/WEB APM services if http metadata exists.
    • Description: Measures the total time spent actually processing for each Service and HTTP status.
    • Metric type: GAUGE
    • Tags: env, service, resource, sublayer_service, http.status_class, and http.status_code.

Duration.by

  • trace.<SPAN_NAME>.duration.by.resource_<2ND_PRIM_TAG>_service.<PERCENTILE_AGGREGATION>:

    • Pre-requisit: This metric exists for any APM service.
    • Description: Measure the total time spent processing by ressource, service, and 2nd primary tag.
    • Metric type: GAUGE
    • Percentile Aggregations: 100p, 50p, 75p, 90p, 95p, 99p
    • Tags: env, service and resource.

  • trace.<SPAN_NAME>.duration.by.resource_service.<PERCENTILE_AGGREGATION>:

    • Pre-requisit: This metric exists for any APM service.
    • Description: Measure the total time spent processing for each ressource and service combination.
    • Metric type: GAUGE
    • Percentile Aggregations: 100p, 50p, 75p, 90p, 95p, 99p
    • Tags: env, service, and resource.

  • trace.<SPAN_NAME>.duration.by.<2ND_PRIM_TAG>_service.<PERCENTILE_AGGREGATION>

    • Pre-requisit: This metric exists for any APM service.
    • Description: Measure the total time spent processing for each 2nd primary tag and service combination.
    • Metric type: GAUGE
    • Percentile Aggregations: 100p, 50p, 75p, 90p, 95p, 99p
    • Tags: env and service.

  • trace.<SPAN_NAME>.duration.by.service.<PERCENTILE_AGGREGATION>

    • Pre-requisit: This metric exists for any APM service.
    • Description: Represents the duration for an individual span. It’s used to track latency and answer questions like, “what’s the median wait time a user experienced?” or “how long do the slowest 1% of users have to wait?”.
    • Metric type: GAUGE
    • Percentile Aggregations: 100p, 50p, 75p, 90p, 95p, 99p
    • Tags: env and service.

Hits

  • trace.<SPAN_NAME>.hits:

    • Pre-requisit: This metric exists for any APM service.
    • Description: Represent the count of hits for a given span.
    • Metric type: COUNT
    • Tags: env, service, resource

  • trace.<SPAN_NAME>.hits.by_http_status

    • Pre-requisit: This metric exists for HTTP/WEB APM services if http metadata exists.
    • Description: Represent the count of hits for a given span break down by HTTP status code.
    • Metric type: COUNT
    • Tags: env, service, resource, http.status_class, and http.status_code.

Span_count

  • trace.<SPAN_NAME>.span_count:

    • Pre-requisit: This metric exists for any APM service.
    • Description: Represent the amount of spans collected on a given interval.
    • Metric type: COUNT
    • Tags: env, service, resource.

  • trace.<SPAN_NAME>.span_count.by_http_status:

    • Pre-requisit: This metric exists for HTTP/WEB APM services if http metadata exists.
    • Description: Represent the amount of spans collected on a given interval break down by HTTP status.
    • Metric type: COUNT
    • Tags: env, service, resource, http.status_class, and http.status_code.

Apdex

  • trace.<SPAN_NAME>.apdex.by.resource_<2ND_PRIM_TAG>_service:

    • Pre-requisit: This metric exists for any HTTP/WEB APM service.
    • Description: Represent the Apdex score for all combination of resources, 2nd primary tags and services.
    • Metric type: GAUGE
    • Tags: env, service, resource

  • trace.<SPAN_NAME>.apdex.by.resource_service:

    • Pre-requisit: This metric exists for any HTTP/WEB APM service.
    • Description: Mesure the Apdex score for each combination of resources and web services.
    • Metric type: GAUGE
    • Tags: env, service, and resource.

  • trace.<SPAN_NAME>.apdex.by.<2ND_PRIM_TAG>_service:

    • Pre-requisit: This metric exists for any HTTP/WEB APM service.
    • Description: Mesure the Apdex score for each combination of 2nd primary tag and web services.
    • Metric type: GAUGE
    • Tags: env and service.

  • trace.<SPAN_NAME>.apdex.by.service:

    • Pre-requisit: This metric exists for any HTTP/WEB APM service.
    • Description: Mesure the Apdex score for each web services.
    • Metric type: GAUGE
    • Tags: env and service.

Further Reading

Additional helpful documentation, links, and articles:

Learn how to setup APM tracing with your applicationDOCUMENTATION more more Discover the list of services reporting to DatadogDOCUMENTATION more more Learn more about services in DatadogDOCUMENTATION more more Dive into your resource performance and tracesDOCUMENTATION more more Understand how to read a Datadog TraceDOCUMENTATION more more
edit Edit

On this Page