Unified Service Tagging
New announcements from Dash: Incident Management, Continuous Profiler, and more! New announcements from Dash!

Unified Service Tagging

Overview

Unified service tagging ties Datadog telemetry together through the use of three reserved tags: env, service, and version.

With these three tags you can:

  • Identify deployment impact with trace and container metrics filtered by version
  • Navigate seamlessly across traces, metrics, and logs with consistent tags
  • View service data based on environment or version in a unified fashion within the Datadog app
Unified Service Tagging

Requirements

  • Unified service tagging requires setup of the Datadog Agent.

  • Unified service tagging requires a tracer version that supports new configurations of the reserved tags. More information can be found per language in the setup instructions.

  • Unified service tagging requires knowledge of configuring tags. If you are unsure of how to configure tags, read the Getting Started with Tagging and Assigning Tags documentation before proceeding to configuration.

Configuration

To begin configuration of unified service tagging, choose your environment:

Containerized environment

In containerized environments, env, service, and version are set through environment variables or standard labels in your Datadog Agent configuration file. Since the agent associates data collected with a specific container, the configuration for these tags can reside within the container’s metadata.

To setup unified service tagging in a containerized environment:

  1. Enable Autodiscovery. This allows the Datadog Agent to automatically identify services running on a specific container and gathers data from those services to map environment variables to the env, service, and version tags.

  2. If you are using Docker, make sure the Agent can access your container’s Docker socket. This allows the Agent detect the environment variables and map them to the standard tags.

  3. Configure your environment based on either full configuration or partial configuration detailed below.

Configuration

Full configuration

To get the full range of unified service tagging when using Kubernetes, add environment variables to both the deployment object level and the pod template spec level:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    tags.datadoghq.com/env: "<ENV>"
    tags.datadoghq.com/service: "<SERVICE>"
    tags.datadoghq.com/version: "<VERSION>"
...
template:
  metadata:
    labels:
      tags.datadoghq.com/env: "<ENV>"
      tags.datadoghq.com/service: "<SERVICE>"
      tags.datadoghq.com/version: "<VERSION>"
  containers:
  -  ...
     env:
          - name: DD_ENV
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['tags.datadoghq.com/env']
          - name: DD_SERVICE
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['tags.datadoghq.com/service']
          - name: DD_VERSION
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['tags.datadoghq.com/version']
Partial configuration
Pod-level metrics

To configure pod-level metrics, add the following standard labels (tags.datadoghq.com) to the pod spec of a Deployment, StatefulSet, or Job:

template:
  metadata:
    labels:
      tags.datadoghq.com/env: "<ENV>"
      tags.datadoghq.com/service: "<SERVICE>"
      tags.datadoghq.com/version: "<VERSION>"

These labels cover pod-level Kubernetes CPU, memory, network, and disk metrics, and can be used for injecting DD_ENV, DD_SERVICE, and DD_VERSION into your service’s container through Kubernetes’s downward API.

If you have multiple containers per pod, you can specify standard labels by container:

tags.datadoghq.com/<container-name>.env
tags.datadoghq.com/<container-name>.service
tags.datadoghq.com/<container-name>.version
State metrics

To configure Kubernetes State Metrics:

  1. Set join_standard_tags to true in your configuration file.

  2. Add the same standard labels to the collection of labels for the parent resource (e.g., Deployment).

    apiVersion: apps/v1
    kind: Deployment
    metadata:
    labels:
      tags.datadoghq.com/env: "<ENV>"
      tags.datadoghq.com/service: "<SERVICE>"
      tags.datadoghq.com/version: "<VERSION>"
    spec:
    template:
      metadata:
        labels:
          tags.datadoghq.com/env: "<ENV>"
          tags.datadoghq.com/service: "<SERVICE>"
          tags.datadoghq.com/version: "<VERSION>"
APM Tracer / StatsD client

To configure APM Tracer and StatsD client environment variables, use the Kubernetes’s downward API in the format below:

containers:
-  ...
    env:
        - name: DD_ENV
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tags.datadoghq.com/env']
        - name: DD_SERVICE
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tags.datadoghq.com/service']
        - name: DD_VERSION
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tags.datadoghq.com/version']
Full configuration

Set the DD_ENV, DD_SERVICE, and DD_VERSION environment variables and corresponding Docker labels for your container to your get the full range of unified service tagging.

The values for service and version can be provided in the Dockerfile:

ENV DD_SERVICE <SERVICE>
ENV DD_VERSION <VERSION>

LABEL com.datadoghq.tags.service="<SERVICE>"
LABEL com.datadoghq.tags.version="<VERSION>"

Since env is likely determined at deploy time, you can inject the environment variable and label later:

docker run -e DD_ENV=<ENV> -l com.datadoghq.tags.env=<ENV> ...

You may also prefer to set everything at deploy time:

docker run -e DD_ENV="<ENV>" \
           -e DD_SERVICE="<SERVICE>" \
           -e DD_VERSION="<VERSION>" \
           -l com.datadoghq.tags.env="<ENV>" \
           -l com.datadoghq.tags.service="<SERVICE>" \
           -l com.datadoghq.tags.version="<VERSION>" \
           ...
Partial configuration

If your service has no need for the Datadog environment variables (for example, third party software like Redis, PostgreSQL, NGINX, and applications not traced by APM) you can just use the Docker labels:

com.datadoghq.tags.env
com.datadoghq.tags.service
com.datadoghq.tags.version

As explained in the full configuration, these labels can be set in a Dockerfile or as arguments for launching the container.

Full configuration

Set the DD_ENV, DD_SERVICE, and DD_VERSION environment variables and corresponding Docker labels in your container’s runtime environment to your get the full range of unified service tagging. For instance, you can set all of this configuration in one place through your ECS task definition:

"environment": [
  {
    "name": "DD_ENV",
    "value": "<ENV>"
  },
  {
    "name": "DD_SERVICE",
    "value": "<SERVICE>"
  },
  {
    "name": "DD_VERSION",
    "value": "<VERSION>"
  }

"dockerLabels": {
  "com.datadoghq.tags.env": "<ENV>",
  "com.datadoghq.tags.service": "<SERVICE>",
  "com.datadoghq.tags.version": "<VERSION>"
  }
]
Partial configuration

If your service has no need for the Datadog environment variables (for example, third party software like Redis, PostgreSQL, NGINX, and applications not traced by APM) you can just use the Docker labels in your ECS task definition:

"dockerLabels": {
  "com.datadoghq.tags.env": "<ENV>",
  "com.datadoghq.tags.service": "<SERVICE>",
  "com.datadoghq.tags.version": "<VERSION>"
}

Non-containerized environment

Depending on how you build and deploy your services’ binaries or executables, you may have several options available for setting environment variables. Since you may run one or more services per host, it is recommended that these environment variables be scoped to a single process.

To form a single point of configuration for all telemetry emitted directly from your service’s runtime for traces, logs, and StatsD metrics, you can either:

  1. Export the environment variables in the command for your executable:

    DD_ENV=<env> DD_SERVICE=<service> DD_VERSION=<version> /bin/my-service

  2. Or use Chef, Ansible, or another orchestration tool to populate a service’s systemd or initd configuration file with the DD environment variables. That way when the service process is started it will have access to those variables.

When configuring your traces for unified service tagging:

  1. Configure the APM Tracer with DD_ENV to keep the definition of env closer to the application that is generating the traces. This method allows the env tag to be sourced automatically from a tag in the span metadata.

  2. Configure spans with DD_VERSION to add version to all spans that fall under the service that belongs to the tracer (generally DD_SERVICE). This means that if your service creates spans with the name of an external service, those spans will not receive version as a tag.

    As long as version is present in spans, it will be added to trace metrics generated from those spans. The version can be added manually in-code or automatically by the APM Tracer. When configured, at the very least these will be used by the APM and Dogstatsd clients to tag trace data and StatsD metrics with env, service, and version. If enabled, the APM tracer will also inject the values of these variables into your logs.

    Note: There can only be one service per span. Trace metrics generally have a single service as well. However, if you have a different service defined in your hosts’ tags, that configured service tag will show up on all trace metrics emitted from that host.

If you’re using connected logs and traces, enable automatic logs injection if supported for your APM Tracer. The APM Tracer will then automatically inject env, service, and version into your logs, thereby eliminating manual configuration for those fields elsewhere.

Note: The PHP Tracer does not support configuration of unified service tagging for logs.

Tags are added in an append-only fashion for custom statsd metrics. For example, if you have two different values for env, the metrics will be tagged with both environments. There is no order in which one tag will override another of the same name.

If your service has access to DD_ENV, DD_SERVICE, and DD_VERSION, then the DogStatsD client will automatically add the corresponding tags to your custom metrics.

Note: The Datadog DogStatsD clients for .NET and PHP do not yet support this functionality.

env and service can also be added to your infrastructure metrics.

The tagging configuration for service metrics lives closer to the Agent in non-containerized contexts. Given that this configuration does not change for each invocation of a service’s process, adding version to the configuration is not recommended.

Single service per host

Set the following configuration in the Agent’s main configuration file:

env: <ENV>
tags:
    - service: <SERVICE>

This setup guarantees consistent tagging of env and service for all data emitted by the Agent.

Multiple services per host

Set the following configuration in the Agent’s main configuration file:

env: <ENV>

To get unique service tags on CPU, memory, and disk I/O metrics at the process level, you can configure a process check:

init_config:
instances:
    - name: web-app
      search_string: ["/bin/web-app"]
      exact_match: false
      service: web-app
    - name: nginx
      search_string: ["nginx"]
      exact_match: false
      service: nginx-web-app

Note: If you already have a service tag set globally in your Agent’s main configuration file, the process metrics will be tagged with two services. Since this can cause confusion with interpreting the metrics, it is recommended to configure the service tag only in the configuration of the process check.

Further Reading