Set Primary Tags to Scope
There are several dimensions available to scope an entire Datadog APM application. These include aggregate statistics (such as requests/second, latency, error rate, Apdex score) and visible traces. These dimensions are set up through primary tags that allow you to get an even finer view of your application’s behavior. Use cases for primary tags include environment, availability zone, datacenter, etc.
Primary tags must follow a different set of rules from those of conventional Datadog tags.
The default and mandatory primary tag is the environment your traces are collected from. Its tag key is
env, and its default value for un-tagged data is
Datadog recommends having the tracer set
env. It also allows for greater flexibility because the definition of
env lives within the actual runtime of the service.
DD_ENV is exposed to your service’s process, the tracer will use it automatically. See Unified Service Tagging to learn about setting
DD_ENV and other standard service environment variables.
You may also manually set
env as a global tag for the tracer in code. See assigning tags in APM for more information.
env tag can be set in your Agent configuration.
Do not set different
env tags on the Tracer and Agent. This may cause duplicate tagging on trace metrics.
Top-level Agent configuration:
Containerized environments: The Agent also supports configuration of the top-level
env through the environment variable
Agent host tag:
Containerized environments: The Agent also supports configuration of top-level
tags through the environment variable
Data by environment
Environments appear at the top of APM pages. Use the
env dropdown to scope the data displayed on the current page.
Add a second primary tag in Datadog
If you need to aggregate your trace metrics across additional dimensions, we recommend setting up a second primary tag in addition to the
default and mandatory primary tag
env:<ENVIRONMENT>. Once configured, a second dropdown is available in the Service Catalog Performance tab.
Go to the APM Settings page to define, change, or remove your primary tags.
- Only organization administrators have access to this page.
- Changes may take up to two hours to be reflected in the UI.
- The tracer always adds
service tags to spans. Datadog recommends never adding these as host level tags to avoid confusion.
- The second primary tag supports up to 30 unique values. See APM Data Volume Guidelines for details.
If you change a previously set primary tag, be aware of the following:
- Historical APM data aggregated by the previously set tag is no longer accessible.
- Any APM monitors scoped to the previous tag display a status of No Data.
You can index your trace metrics based on the tags derived from Docker containers and Kubernetes pod metadata on Linux-based platforms. Container-based second primary tags are available in Datadog Agent versions 7.35.0 and later.
To enable container-based second primary tags, install Agent version 7.35.0 or later, update the CID stats setting as described below, and restart the Agent. The procedure for enabling depends on how you installed the Agent:
Using the Datadog Helm chart version 2.26.2 or later, add the following to your values file:
- name: DD_APM_FEATURES
Use the following environment variable in the Agent DaemonSet. If you are running a container per Agent process, add the following environment variable to all containers. Otherwise, add it to the Agent container.
- name: DD_APM_FEATURES
Add the following to your docker-compose.yml file:
- DD_APM_FEATURES: 'enable_cid_stats'
If you configure the Agent with environment variables, as is common with Docker and ECS installations, pass the following environment variable to the trace Agent after upgrading the Docker image.
Restart the Agent. Go to the APM Settings page and select the second primary tag you want to use. It can take up to two hours for changes to this setting to take effect.
Now you can filter your services in the Service List by the tag being sent by your containerized services. Trace metrics used by Dashboards and Monitors can also be aggregated by the container primary tag.
If you haven’t already, you may also configure the Agent to send container or Pod labels as custom tags for your traces with Assigning Tags.
View data by primary tag
Primary tags appear at the top of APM pages. Use these selectors to filter the data displayed on the current page. To view all data independent of a primary tag, choose
<TAG_NAME>:* from the dropdown.
Additional helpful documentation, links, and articles: