A service can handle a variety of requests, some of which you might not want traced or included in trace metrics. An example of this is, possibly, health checks in a web application.
There are two ways to specify that such an endpoint should be untraced and excluded from trace metrics:
Note: Filtering traces using any of the following options removes these requests from trace metrics. For information on how to reduce ingestion without affecting the trace metrics, see ingestion controls.
The Trace Agent component within the Datadog Agent has two methods to prevent certain traces from coming through: ignoring span tags or ignoring resources. If traces are dropped due to these settings, the trace metrics exclude these requests.
Configuring the Trace Agent to ignore certain spans or resources applies to all services that send traces to this particular Datadog Agent. If you have application-specific requirements, use the Tracer configuration method instead.
Ignoring based on span tags
Starting with Datadog Agent 6.27.0/7.27.0, the filter tags option drops traces with root spans that match specified span tags. This option applies to all services that send traces to this particular Datadog Agent. Traces that are dropped because of filter tags are not included in trace metrics.
If you can programmatically identify a set of traces that you know you don’t want to send to Datadog, and no other option in this guide solves your requirement, you can consider adding a custom span tag so you can drop the traces. Reach out to Support to discuss your use case further so Datadog can continue to expand this functionality.
The filter tags option requires an exact string match. If your use case requires ignoring by regex, see Ignoring based on resources.
You can specify span tags to require or reject by using a list of keys and values separated by spaces in environment variables:
DD_APM_FILTER_TAGS_REQUIRE
Collects only traces that have root spans with an exact match for the specified span tags and values. If it does not match this rule, the trace is dropped. For example, DD_APM_FILTER_TAGS_REQUIRE="key1:value1 key2:value2". In Datadog Agent 7.49+, regular expressions can be provided with DD_APM_FILTER_TAGS_REGEX_REQUIRE.
DD_APM_FILTER_TAGS_REJECT
Rejects traces that have root spans with an exact match for the specified span tags and values. If it matches this rule, the trace is dropped. For example, DD_APM_FILTER_TAGS_REJECT="key1:value1 key2:value2". In Datadog Agent 7.49+, regular expressions can be provided with DD_APM_FILTER_TAGS_REGEX_REJECT.
Alternatively, you can set them in the Agent configuration with a comma-separated list:
In the traceAgent section of the values.yaml file, add DD_APM_FILTER_TAGS_REJECT in the env section, then spin up helm as usual. For multiple tags, separate each key:value with a space.
values.yaml
traceAgent:# agents.containers.traceAgent.env -- Additional environment variables for the trace-agent containerenv:- name:DD_APM_FILTER_TAGS_REJECTvalue:tag_key1:tag_val2 tag_key2:tag_val2
Filtering traces this way removes these requests from trace metrics. For more information on how to reduce ingestion without affecting the trace metrics, see Ingestion Controls.
On the backend, Datadog creates and adds the following span tags to spans after ingestion. These tags cannot be used to drop traces at the Datadog Agent level.
Name
Description
http.path_group
The full URL path from the http.url tag.
http.url_details.host
The host name portion of the http.url tag.
http.url_details.path
The full request target as passed in an HTTP request line or equivalent.
http.url_details.scheme
The request scheme from the http.url tag.
http.url_details.queryString
The query string portion from the http.url tag.
http.url_details.port
The HTTP port from the http.url tag.
http.useragent_details.os.family
The OS family reported by the User-Agent.
http.useragent_details.browser.family
The browser family reported by the User-Agent.
http.useragent_details.device.family
The device family reported by the User-Agent.
Note: Starting from October 1st 2022, Datadog backend applies a remapping in order to apply Span Tags Semantics across tracers on all ingested spans. If you want to drop spans based on tags at the Datadog Agent level, use tags in the Remap from column.
Network communications
Name
Remap from
network.host.ip
tcp.local.address - Node.js
network.destination.ip
out.host - All languages
network.destination.port
grpc.port - Python tcp.remote.port - Node.js out.port - All languages
grpc.request.metadata.* - Python, Node.js rpc.grpc.request.metadata - Go
rpc.grpc.response.metadata.*
grpc.response.metadata.* - Python, Node.js
Errors
Name
Remap from
error.message
error.msg - All languages
Ignoring based on resources
The ignore resources option allows resources to be excluded if the global root span of the trace matches certain criteria. See Exclude resources from being collected. This option applies to all services that send traces to this particular Datadog Agent. Traces that are dropped because of ignore resources are not included in trace metrics.
You can specify resources to ignore either in the Agent configuration file, datadog.yaml, or with the DD_APM_IGNORE_RESOURCES environment variable. See examples below.
datadog.yaml
apm_config:## @param ignore_resources - list of strings - optional## A list of regular expressions can be provided to exclude certain traces based on their resource name.## All entries must be surrounded by double quotes and separated by commas.ignore_resources:["(GET|POST) /healthcheck","API::NotesController#index"]
Notes:
The regex syntax that the Trace Agent accepts is evaluated by Go’s regexp.
Depending on your deployment strategy, you may have to adjust the regex by escaping special characters.
If you use dedicated containers with Kubernetes, make sure that the environment variable for the ignore resource option is being applied to the trace-agent container.
Example
Consider a trace that contains calls to /api/healthcheck that you don’t want traces from:
Take note of the resource name of the global root span.
Operation name: rack.request
Resource name: Api::HealthchecksController#index
Http.url: /api/healthcheck
To use the ignore resource option correctly, the regex rule written must match with the resource name, Api::HealthchecksController#index. A few regex options are possible, but to filter out traces from this resource exactly as is, a potential regex to use is Api::HealthchecksController#index$.
Depending on how you deploy, the syntax looks a little different:
In the Datadog Agent container’s list of environment variables, add DD_APM_IGNORE_RESOURCES with a pattern like the example below. Docker Compose has its own variable substitution to consider when you use special characters like $.
environment:// other Datadog Agent environment variables- DD_APM_IGNORE_RESOURCES=Api::HealthchecksController#index$$
For multiple values:
environment:// other Datadog Agent environment variables- DD_APM_IGNORE_RESOURCES="value1","Api::HealthchecksController#index$$"
In your docker run command to spin up the Datadog Agent, add DD_APM_IGNORE_RESOURCES:
If you use Amazon ECS (such as on EC2), in your Datadog Agent container definition, add the environment variable DD_APM_IGNORE_RESOURCES with the values such that the JSON evaluates to something like this:
"environment":[// other environment variables for the Datadog Agent
{"name":"DD_APM_IGNORE_RESOURCES","value":"Api::HealthchecksController#index$"}]
Note: Filtering traces this way removes these requests from trace metrics. For information on how to reduce ingestion without affecting the trace metrics, see ingestion controls.
Tracer configuration options
Some of the language-specific tracers have an option to modify spans before they are sent to the Datadog Agent. Use this option if you have application-specific requirements and are using a language listed below.
Important: If the request is associated with a distributed trace, the resulting trace can have sampling inaccuracy if you drop portions of it due to these filtering rules.
The Ruby tracer has a post-processing pipeline that removes traces that meet certain criteria. More information and examples can be found in Post-processing traces.
For example, if the resource name is Api::HealthchecksController#index, use the Datadog::Tracing::Pipeline::SpanFilter class to remove traces that contain the resource name. This filter can also be used to match on other metadata available for the span object.
The Python tracer has a FilterRequestsOnUrl filter you can configure to remove traces from certain endpoints. Alternatively, you can write a custom filter. See Trace Filtering for more information.
Suppose the root span’s http.url span tag has a value of http://<domain>/healthcheck. Use the following regex to match against any endpoint ending in healthcheck:
from ddtrace import tracer
from ddtrace.filters import FilterRequestsOnUrl
tracer.configure(settings={
'FILTERS': [
FilterRequestsOnUrl(r'http://.*/healthcheck$'),
],
})
Configure a blocklist on the Http plugin. Take note of what the blocklist matches on from the API docs. For example, Http matches on URLs, so if the trace’s http.url span tag is http://<domain>/healthcheck, write a rule that matches the healthcheck URL:
Note: The tracer configuration for the integration must come before that instrumented module is imported.
The Java tracer has an option for a custom TraceInterceptor to filter out certain spans. See Extending Tracers.
For example, if your resource name is GET /healthcheck, write a trace interceptor that drops traces containing this resource name. Adjust the logic to meet your use case.
public class GreetingController {
static {
// In a class static block to avoid initializing multiple times.
GlobalTracer.get().addTraceInterceptor(new TraceInterceptor() {
@Override
public Collection<? extends MutableSpan> onTraceComplete(Collection<? extends MutableSpan> trace) {
for (MutableSpan span : trace) {
if ("GET /healthcheck".contentEquals(span.getResourceName())) {
return Collections.emptyList();
}
}
return trace;
}
@Override
public int priority() {
return 200; // Some unique number
}
});
}
}
Note: Filtering traces this way removes these requests from trace metrics. For information on how to reduce ingestion without affecting the trace metrics, see ingestion controls.
