---
title: Ingestion Mechanisms
description: >-
  Overview of the mechanisms in the SDK and the Agent that control trace
  ingestion.
breadcrumbs: Docs > APM > The Trace Pipeline > Ingestion Mechanisms
---

# Ingestion Mechanisms

{% image
   source="https://docs.dd-static.net/images/tracing/apm_lifecycle/ingestion_sampling_rules.049c4637df137191e7beb433e16173e6.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/tracing/apm_lifecycle/ingestion_sampling_rules.049c4637df137191e7beb433e16173e6.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Ingestion Sampling Rules" /%}

Multiple mechanisms determine whether spans generated by your applications are sent to Datadog (*ingested*). The logic behind these mechanisms is in the [SDKs](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries.md) and in the Datadog Agent. Depending on the configuration, all or some of the traffic generated by instrumented services is ingested.

Each ingested span has a unique **ingestion reason** that refers to one of the mechanisms described on this page. The [usage metrics](https://docs.datadoghq.com/tracing/trace_pipeline/metrics.md) `datadog.estimated_usage.apm.ingested_bytes` and `datadog.estimated_usage.apm.ingested_spans` are tagged by `ingestion_reason`.

Use the [Ingestion Reasons dashboard](https://app.datadoghq.com/dash/integration/apm_ingestion_reasons) to investigate each ingestion reason in context and identify which configuration options to focus on.

## Head-based sampling{% #head-based-sampling %}

The default sampling mechanism is called *head-based sampling*. The decision to keep or drop a trace is made at the start of the [root span](https://docs.datadoghq.com/tracing/glossary.md#trace-root-span) and then propagated to other services as part of their request context (for example, as an HTTP request header).

Because the decision is made at the beginning of the trace and conveyed to all parts, the trace is kept or dropped as a whole.

{% image
   source="https://docs.dd-static.net/images/tracing/guide/ingestion_sampling_use_cases/head-based-sampling.de2525a2b984351fb254ade66855bbe6.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/tracing/guide/ingestion_sampling_use_cases/head-based-sampling.de2525a2b984351fb254ade66855bbe6.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Head-based Sampling" /%}

You can set sampling rates for head-based sampling in two places:

- At the **Agent** level (default)
- At the **SDK** level: any SDK mechanism overrides the Agent setup.

### In the Agent{% #in-the-agent %}

`ingestion_reason: auto`

The Datadog Agent continuously sends sampling rates to SDKs to apply at the root of traces. The Agent adjusts rates to achieve a target of ten traces per second overall, distributed to services depending on the traffic.

For instance, if service `A` has more traffic than service `B`, the Agent might vary the sampling rate for `A` such that `A` keeps no more than seven traces per second, and similarly adjust the sampling rate for `B` such that `B` keeps no more than three traces per second, for a total of 10 traces per second.

#### Remote configuration{% #remote-configuration %}

Sampling rate configuration in the Agent is configurable remotely if you are using Agent version [7.42.0](https://github.com/DataDog/datadog-agent/releases/tag/7.42.0) or higher. To get started, set up [Remote Configuration](https://docs.datadoghq.com/tracing/guide/remote_config.md) and then configure the `ingestion_reason` parameter from the [Ingestion Control page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md). Remote Configuration allows you to change the parameter without restarting the Agent. Remotely set configuration takes precedence over local configurations, including environment variables and settings from `datadog.yaml`.

#### Local configuration{% #local-configuration %}

Set the Agent's target traces-per-second in its main configuration file (`datadog.yaml`) or as an environment variable:

```
@param target_traces_per_second - integer - optional - default: 10
@env DD_APM_TARGET_TPS - integer - optional - default: 10
```

**Notes**:

- The traces-per-second sampling rate set in the Agent only applies to Datadog SDKs. It has no effect on other SDKs such as OpenTelemetry SDKs.
- The target is not a fixed value. In reality, it fluctuates depending on traffic spikes and other factors.

Spans from traces sampled by the Datadog Agent's automatic sampling rates are tagged with ingestion reason `auto`. The `ingestion_reason` tag is also set on [usage metrics](https://docs.datadoghq.com/tracing/trace_pipeline/metrics.md). Services using this default mechanism are labeled `Automatic` in the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md) Configuration column.

### In SDKs: user-defined rules{% #in-sdks-user-defined-rules %}

`ingestion_reason: rule`

For more granular control, use SDK sampling configuration options:

- Set a specific **sampling rate to apply to the root of the trace** by service or resource name, overriding the Agent's default mechanism.
- Set a **rate limit** on the number of ingested traces per second. The default rate limit is 100 traces per second per service instance. When using the Agent default mechanism, the rate limiter is ignored.

**Note**: Sampling rules are also head-based sampling controls. If the traffic for a service is higher than the configured maximum traces per second, then traces are dropped at the root. It does not create incomplete traces.

The configuration can be set by environment variables or directly in the code:

{% tab title="Java" %}
**Remote configuration**

Starting from version [1.34.0](https://github.com/DataDog/dd-trace-java/releases/tag/v1.34.0), for Java applications, set by-service and by-resource sampling rates from the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md#configure-the-service-ingestion-rate) UI.

Read more about how to remotely configure sampling rates by service and resource in the [Resource-based sampling guide](https://docs.datadoghq.com/tracing/guide/resource_based_sampling.md).

**Note**: Remotely set configuration takes precedence over local configuration.

**Local configuration**

For Java applications, set by-service and by-resource (starting from version [v1.26.0](https://github.com/DataDog/dd-trace-java/releases/tag/v1.26.0) for resource-based sampling) sampling rates with the `DD_TRACE_SAMPLING_RULES` environment variable.

For example, to capture 100% of traces for the resource `GET /checkout` from the service `my-service`, and 20% of other endpoints' traces, set:

```gdscript3
# using system property
java -Ddd.trace.sampling.rules='[{"service": "my-service", "resource": "GET /checkout", "sample_rate":1},{"service": "my-service", "sample_rate":0.2}]' -javaagent:dd-java-agent.jar -jar my-app.jar

# using environment variables
export DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "resource":"GET /checkout", "sample_rate": 1},{"service": "my-service", "sample_rate": 0.2}]'
```

The service name value is case sensitive and must match the case of the actual service name.

Configure a rate limit by setting the `DD_TRACE_RATE_LIMIT` environment variable to the maximum traces per second per service instance. If no `DD_TRACE_RATE_LIMIT` value is set, a limit of 100 traces per second is applied.

**Note**: The use of `DD_TRACE_SAMPLE_RATE` is deprecated. Use `DD_TRACE_SAMPLING_RULES` instead. For instance, if you already set `DD_TRACE_SAMPLE_RATE` to `0.1`, set `DD_TRACE_SAMPLING_RULES` to `[{"sample_rate":0.1}]` instead.

Read more about sampling controls in the [Java SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/java.md).
{% /tab %}

{% tab title="Python" %}
**Remote configuration**

Starting from version [2.9.0](https://github.com/DataDog/dd-trace-py/releases/tag/v2.9.0), for Python applications, set by-service and by-resource sampling rates from the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md#configure-the-service-ingestion-rate) UI.

Read more about how to remotely configure sampling rates by service and resource in the [Resource-based sampling guide](https://docs.datadoghq.com/tracing/guide/resource_based_sampling.md).

**Note**: Remotely set configuration takes precedence over local configuration.

**Local configuration** For Python applications, set by-service and by-resource (starting from version [v2.8.0](https://github.com/DataDog/dd-trace-py/releases/tag/v2.8.0) for resource-based sampling) sampling rates with the `DD_TRACE_SAMPLING_RULES` environment variable.

For example, to capture 100% of traces for the resource `GET /checkout` from the service `my-service`, and 20% of other endpoints' traces, set:

```gdscript3
export DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "resource": "GET /checkout", "sample_rate": 1},{"service": "my-service", "sample_rate": 0.2}]'
```

Configure a rate limit by setting the `DD_TRACE_RATE_LIMIT` environment variable to the maximum traces per second per service instance. If no `DD_TRACE_RATE_LIMIT` value is set, a limit of 100 traces per second is applied.

**Note**: The use of `DD_TRACE_SAMPLE_RATE` is deprecated. Use `DD_TRACE_SAMPLING_RULES` instead. For instance, if you already set `DD_TRACE_SAMPLE_RATE` to `0.1`, set `DD_TRACE_SAMPLING_RULES` to `[{"sample_rate":0.1}]` instead.

Read more about sampling controls in the [Python SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/python.md).
{% /tab %}

{% tab title="Ruby" %}
**Remote configuration**

Starting from version [2.0.0](https://github.com/DataDog/dd-trace-rb/releases/tag/v2.0.0), for Ruby applications, set by-service and by-resource sampling rates from the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md#configure-the-service-ingestion-rate) UI.

Read more about how to remotely configure sampling rates by service and resource in the [Resource-based sampling guide](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/ruby.md#sampling).

**Note**: Remotely set configuration takes precedence over local configuration.

**Local configuration** For Ruby applications, set a global sampling rate for the library using the `DD_TRACE_SAMPLE_RATE` environment variable. Set by-service sampling rates with the `DD_TRACE_SAMPLING_RULES` environment variable.

For example, to send 50% of the traces for the service named `my-service` and 10% of the rest of the traces:

```gdscript3
export DD_TRACE_SAMPLE_RATE=0.1
export DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "sample_rate": 0.5}]'
```

Configure a rate limit by setting the `DD_TRACE_RATE_LIMIT` environment variable to the maximum traces per second per service instance. If no `DD_TRACE_RATE_LIMIT` value is set, a limit of 100 traces per second is applied.

Read more about sampling controls in the [Ruby SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/ruby.md#sampling).
{% /tab %}

{% tab title="Go" %}
**Remote configuration**

Starting from version [1.64.0](https://github.com/DataDog/dd-trace-go/releases/tag/v1.64.0), for Go applications, set by-service and by-resource sampling rates from the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md#configure-the-service-ingestion-rate) UI.

Read more about how to remotely configure sampling rates by service and resource in this [article](https://docs.datadoghq.com/tracing/guide/resource_based_sampling.md).

**Note**: The remotely set configuration takes precedence over local configuration.

**Local configuration**

For Go applications, set by-service and by-resource (starting from version [v1.60.0](https://github.com/DataDog/dd-trace-go/releases/tag/v1.60.0) for resource-based sampling) sampling rates with the `DD_TRACE_SAMPLING_RULES` environment variable.

For example, to capture 100% of traces for the resource `GET /checkout` from the service `my-service`, and 20% of other endpoints' traces, set:

```gdscript3
export DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "resource": "GET /checkout", "sample_rate": 1},{"service": "my-service", "sample_rate": 0.2}]'
```

Configure a rate limit by setting the `DD_TRACE_RATE_LIMIT` environment variable to the maximum traces per second per service instance. If no `DD_TRACE_RATE_LIMIT` value is set, a limit of 100 traces per second is applied.

**Note**: The use of `DD_TRACE_SAMPLE_RATE` is deprecated. Use `DD_TRACE_SAMPLING_RULES` instead. For instance, if you already set `DD_TRACE_SAMPLE_RATE` to `0.1`, set `DD_TRACE_SAMPLING_RULES` to `[{"sample_rate":0.1}]` instead.

Read more about sampling controls in the [Go SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/go.md).
{% /tab %}

{% tab title="Node.js" %}
**Remote configuration**

Starting from version [5.16.0](https://github.com/DataDog/dd-trace-js/releases/tag/v5.16.0), for Node.js applications, set by-service and by-resource sampling rates from the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md#configure-the-service-ingestion-rate) UI.

Read more about how to remotely configure sampling rates by service and resource in the [Resource-based sampling guide](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/nodejs.md).

**Note**: Remotely set configuration takes precedence over local configuration.

**Local configuration**

For Node.js applications, set a global sampling rate in the library using the `DD_TRACE_SAMPLE_RATE` environment variable.

You can also set by-service sampling rates. For instance, to send 50% of the traces for the service named `my-service` and 10% for the rest of the traces:

```javascript
tracer.init({
    ingestion: {
        sampler: {
            sampleRate: 0.1,
            rules: [
                { sampleRate: 0.5, service: 'my-service' }
            ]
        }
    }
});
```

Configure a rate limit by setting the `DD_TRACE_RATE_LIMIT` environment variable to the maximum traces per second per service instance. If no `DD_TRACE_RATE_LIMIT` value is set, a limit of 100 traces per second is applied.

Read more about sampling controls in the [Node.js SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/nodejs.md).
{% /tab %}

{% tab title="PHP" %}
**Remote configuration**

Starting from version [1.4.0](https://github.com/DataDog/dd-trace-php/releases/tag/1.4.0), for PHP applications, set by-service and by-resource sampling rates from the [Ingestion Control Page](https://app.datadoghq.com/apm/traces/ingestion-control).

Read more about how to remotely configure sampling rates by service and resource in the [Resource-based sampling guide](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/php.md).

**Note**: Remotely-set configuration takes precedence over local configuration.

**Local configuration**

For PHP applications, set a global sampling rate for the library using the `DD_TRACE_SAMPLE_RATE` environment variable. Set by-service sampling rates with the `DD_TRACE_SAMPLING_RULES` environment variable.

For example, to send 50% of the traces for the service named `my-service`, 20% of other endpoints' traces, and 10% for the rest of the traces, set:

```gdscript3
export DD_TRACE_SAMPLE_RATE=0.1
export DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "resource":"GET /checkout", "sample_rate": 1},{"service": "my-service", "sample_rate": 0.2}]'
```

Read more about sampling controls in the [PHP SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/php.md).
{% /tab %}

{% tab title="C++" %}
**Remote configuration**

Starting from version [0.2.2](https://github.com/DataDog/dd-trace-cpp/releases/tag/v0.2.2), for C++ applications, set by-service and by-resource sampling rates from the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md#configure-the-service-ingestion-rate) UI.

Read more about how to remotely configure sampling rates by service and resource in the [Resource-based sampling guide](https://github.com/DataDog/dd-trace-cpp/releases/tag/v0.1.0).

**Note**: Remotely set configuration takes precedence over local configuration.

**Local configuration** Starting in [v0.1.0](https://github.com/DataDog/dd-trace-cpp/releases/tag/v0.1.0), the Datadog C++ library supports the following configurations:

- Global sampling rate: `DD_TRACE_SAMPLE_RATE` environment variable
- Sampling rates by service: `DD_TRACE_SAMPLING_RULES` environment variable.
- Rate limit setting: `DD_TRACE_RATE_LIMIT` environment variable.

For example, to send 50% of the traces for the service named `my-service` and 10% for the rest of the traces:

```gdscript3
export DD_TRACE_SAMPLE_RATE=0.1
export DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "sample_rate": 0.5}]'
```

C++ does not provide integrations for automatic instrumentation, but it's used by proxy tracing such as Envoy, NGINX, or Istio. Read more about how to configure sampling for proxies in [Tracing proxies](https://docs.datadoghq.com/tracing/trace_collection/proxy_setup.md).
{% /tab %}

{% tab title=".NET" %}
For .NET applications, set a global sampling rate for the library using the `DD_TRACE_SAMPLE_RATE` environment variable. Set by-service sampling rates with the `DD_TRACE_SAMPLING_RULES` environment variable.

For example, to send 50% of the traces for the service named `my-service` and 10% for the rest of the traces:

```
#using powershell
$env:DD_TRACE_SAMPLE_RATE=0.1
$env:DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "sample_rate": 0.5}]'

#using JSON file   
{
    "DD_TRACE_SAMPLE_RATE": "0.1",
    "DD_TRACE_SAMPLING_RULES": "[{\"service\": \"my-service\", \"resource\": \"GET /checkout\", \"sample_rate\": 0.5}]"
}
```

{% alert level="info" %}
Starting in version 2.35.0, if [Agent Remote Configuration](https://docs.datadoghq.com/remote_configuration.md) is enabled where the service runs, you can set a per-service `DD_TRACE_SAMPLE_RATE` in the [Software Catalog](https://docs.datadoghq.com/tracing/software_catalog.md) UI.
{% /alert %}

Configure a rate limit by setting the `DD_TRACE_RATE_LIMIT` environment variable to the maximum traces per second per service instance. If no `DD_TRACE_RATE_LIMIT` value is set, a limit of 100 traces per second is applied.

Read more about sampling controls in the [.NET SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/dotnet-core.md).Read more about [configuring environment variables for .NET](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/dotnet-core.md?tab=registryeditor#configuring-process-environment-variables).
{% /tab %}

**Note**: All the spans from a trace sampled using an SDK configuration are tagged with the ingestion reason `rule`. Services configured with user-defined sampling rules are marked as `Configured` in the [Ingestion Control Page](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md) Configuration column.

## Error and rare traces{% #error-and-rare-traces %}

For traces not caught by head-based sampling, two additional Datadog Agent sampling mechanisms catch critical and diverse traces that would otherwise be dropped. These samplers keep a diverse set of local traces (spans from the same host) by catching all combinations of a predetermined set of tags:

- **Error traces**: Error sampling provides visibility into potential system failures.
- **Rare traces**: Rare trace sampling maintains visibility into low-traffic services and resources across your system.

**Note**: Error and rare samplers are ignored for services for which you set library sampling rules.

### Error traces{% #error-traces %}

`ingestion_reason: error`

The error sampler catches pieces of traces that contain error spans not caught by head-based sampling, at a rate of up to 10 traces per second per Agent. This helps maintain visibility into errors when the head-based sampling rate is low.

With Agent version 7.33 and forward, you can configure the error sampler in the Agent main configuration file (`datadog.yaml`) or with environment variables:

```
@param errors_per_second - integer - optional - default: 10
@env DD_APM_ERROR_TPS - integer - optional - default: 10
```

{% image
   source="https://docs.dd-static.net/images/tracing/guide/ingestion_sampling_use_cases/error-spans-sampling.ed363923534c6d5bfde5599d8f7dc10e.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/tracing/guide/ingestion_sampling_use_cases/error-spans-sampling.ed363923534c6d5bfde5599d8f7dc10e.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Error Sampling" /%}

**Notes**:

1. Set the parameter to `0` to disable the error sampler.
1. The error sampler captures local error traces at the Agent level. If the trace is distributed, the complete trace may not be sent to Datadog.
1. By default, spans dropped by SDK rules or custom logic such as `manual.drop` are **excluded** under the error sampler.

#### Datadog Agent 7.42.0 and higher{% #datadog-agent-7420-and-higher %}

Error sampling is remotely configurable if you're using Agent version [7.42.0](https://github.com/DataDog/datadog-agent/releases/tag/7.42.0) or higher. Follow the [documentation](https://docs.datadoghq.com/tracing/guide/remote_config.md) to enable remote configuration in your Agents. With remote configuration, you can enable the collection of rare spans without restarting the Datadog Agent.

#### Datadog Agent 6/7.41.0 and higher{% #datadog-agent-67410-and-higher %}

To override the default behavior so that spans dropped by the SDK rules or custom logic such as `manual.drop` are **included** by the error sampler, enable the feature with: `DD_APM_FEATURES=error_rare_sample_tracer_drop` in the Datadog Agent (or the dedicated Trace Agent container within the Datadog Agent pod in Kubernetes).

#### Datadog Agent 6/7.33 to 6/7.40.x{% #datadog-agent-6733-to-6740x %}

Error sampling default behavior can't be changed for these Agent versions. Upgrade the Datadog Agent to Datadog Agent 6/7.41.0 and higher.

### Rare traces{% #rare-traces %}

`ingestion_reason: rare`

The rare sampler sends a set of rare spans to Datadog. It catches combinations of `env`, `service`, `name`, `resource`, `error.type`, and `http.status` at up to 5 traces per second per Agent. This helps maintain visibility into low-traffic resources when the head-based sampling rate is low.

**Note**: The rare sampler captures local traces at the Agent level. If the trace is distributed, there is no guarantee that the complete trace is sent to Datadog.

#### Datadog Agent 7.42.0 and higher{% #datadog-agent-7420-and-higher-1 %}

Rare sampling is remotely configurable if you're using Agent version [7.42.0](https://github.com/DataDog/datadog-agent/releases/tag/7.42.0) or higher. Follow the [documentation](https://docs.datadoghq.com/tracing/guide/remote_config.md) to enable remote configuration in your Agents. With remote configuration, you can change the parameter value without restarting the Datadog Agent.

#### Datadog Agent 6/7.41.0 and higher{% #datadog-agent-67410-and-higher-1 %}

By default, the rare sampler is **not enabled**.

**Note**: When **enabled**, spans dropped by SDK rules or custom logic such as `manual.drop` are **excluded** under this sampler.

To configure the rare sampler, update the `apm_config.enable_rare_sampler` setting in the Agent main configuration file (`datadog.yaml`) or with the environment variable `DD_APM_ENABLE_RARE_SAMPLER`:

```
@params apm_config.enable_rare_sampler - boolean - optional - default: false
@env DD_APM_ENABLE_RARE_SAMPLER - boolean - optional - default: false
```

To evaluate spans dropped by SDK rules or custom logic such as `manual.drop`, enable the feature with: `DD_APM_FEATURES=error_rare_sample_tracer_drop` in the Trace Agent.

#### Datadog Agent 6/7.33 to 6/7.40.x{% #datadog-agent-6733-to-6740x-1 %}

By default, the rare sampler is enabled.

**Note**: When **enabled**, spans dropped by SDK rules or custom logic such as `manual.drop` **are excluded** under this sampler. To include these spans in this logic, upgrade to Datadog Agent 6.41.0/7.41.0 or higher.

To change the default rare sampler settings, update the `apm_config.disable_rare_sampler` setting in the Agent main configuration file (`datadog.yaml`) or with the environment variable `DD_APM_DISABLE_RARE_SAMPLER`:

```
@params apm_config.disable_rare_sampler - boolean - optional - default: false
@env DD_APM_DISABLE_RARE_SAMPLER - boolean - optional - default: false
```

## Force keep and drop{% #force-keep-and-drop %}

`ingestion_reason: manual`

The head-based sampling mechanism can be overridden at the SDK level. For example, if you need to monitor a critical transaction, you can force the associated trace to be kept. On the other hand, for unnecessary or repetitive information like health checks, you can force the trace to be dropped.

- Set Manual Keep on a span to indicate that it and all child spans should be ingested. The resulting trace might appear incomplete in the UI if the span in question is not the root span of the trace.

- Set Manual Drop on a span to make sure that **no** child span is ingested. The error and rare samplers are ignored in the Agent.

{% tab title="Java" %}
Manually keep a trace:

```java
import datadog.trace.api.DDTags;
import io.opentracing.Span;
import datadog.trace.api.Trace;
import io.opentracing.util.GlobalTracer;

public class MyClass {
    @Trace
    public static void myMethod() {
        // grab the active span out of the traced method
        Span span = GlobalTracer.get().activeSpan();
        // Always keep the trace
        span.setTag(DDTags.MANUAL_KEEP, true);
        // method impl follows
    }
}
```

Manually drop a trace:

```java
import datadog.trace.api.DDTags;
import io.opentracing.Span;
import datadog.trace.api.Trace;
import io.opentracing.util.GlobalTracer;

public class MyClass {
    @Trace
    public static void myMethod() {
        // grab the active span out of the traced method
        Span span = GlobalTracer.get().activeSpan();
        // Always Drop the trace
        span.setTag(DDTags.MANUAL_DROP, true);
        // method impl follows
    }
}
```

{% /tab %}

{% tab title="Python" %}
Manually keep a trace:

```python
from ddtrace import tracer
from ddtrace.constants import MANUAL_DROP_KEY, MANUAL_KEEP_KEY

@tracer.wrap()
def handler():
    span = tracer.current_span()
    # Always Keep the Trace
    span.set_tag(MANUAL_KEEP_KEY)
    # method impl follows
```

Manually drop a trace:

```python
from ddtrace import tracer
from ddtrace.constants import MANUAL_DROP_KEY, MANUAL_KEEP_KEY

@tracer.wrap()
def handler():
    span = tracer.current_span()
    # Always Drop the Trace
    span.set_tag(MANUAL_DROP_KEY)
    # method impl follows
```

{% /tab %}

{% tab title="Ruby" %}
Manually keep a trace:

```ruby
Datadog::Tracing.trace(name, options) do |span, trace|
  trace.keep! # Affects the active trace
  # Method implementation follows
end
```

Manually drop a trace:

```ruby
Datadog::Tracing.trace(name, options) do |span, trace|
  trace.reject! # Affects the active trace
  # Method implementation follows
end
```

{% /tab %}

{% tab title="Go" %}
**Note**: This documentation uses v2 of the Go tracer, which Datadog recommends for all users. If you are using v1, see the [migration guide](https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/go/migration.md) to upgrade to v2.

Manually keep a trace:

```Go
package main

import (
    "log"
    "net/http"
    "github.com/DataDog/dd-trace-go/v2/ddtrace/ext" 
    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Always keep this trace:
    span.SetTag(ext.ManualKeep, true)
    //method impl follows

}
```

Manually drop a trace:

```Go
package main

import (
    "log"
    "net/http"

    "github.com/DataDog/dd-trace-go/v2/ddtrace/ext"
    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Always drop this trace:
    span.SetTag(ext.ManualDrop, true)
    //method impl follows
}
```

{% /tab %}

{% tab title="Node.js" %}
Manually keep a trace:

```js
const tracer = require('dd-trace')
const tags = require('dd-trace/ext/tags')

const span = tracer.startSpan('web.request')

// Always keep the trace
span.setTag(tags.MANUAL_KEEP)
//method impl follows
```

Manually drop a trace:

```js
const tracer = require('dd-trace')
const tags = require('dd-trace/ext/tags')

const span = tracer.startSpan('web.request')

// Always drop the trace
span.setTag(tags.MANUAL_DROP)
//method impl follows
```

{% /tab %}

{% tab title=".NET" %}
Manually keep a trace:

```cs
using Datadog.Trace;

using(var scope = Tracer.Instance.StartActive("my-operation"))
{
    var span = scope.Span;

    // Always keep this trace
    span.SetTag(Datadog.Trace.Tags.ManualKeep, "true");
    //method impl follows
}
```

Manually drop a trace:

```cs
using Datadog.Trace;

using(var scope = Tracer.Instance.StartActive("my-operation"))
{
    var span = scope.Span;

    // Always drop this trace
    span.SetTag(Datadog.Trace.Tags.ManualDrop, "true");
    //method impl follows
}
```

{% /tab %}

{% tab title="PHP" %}
Manually keep a trace:

```php
<?php
  $tracer = \DDTrace\GlobalTracer::get();
  $span = $tracer->getActiveSpan();

  if (null !== $span) {
    // Always keep this trace
    $span->setTag(\DDTrace\Tag::MANUAL_KEEP, true);
  }
?>
```

Manually drop a trace:

```php
<?php
  $tracer = \DDTrace\GlobalTracer::get();
  $span = $tracer->getActiveSpan();

  if (null !== $span) {
    // Always drop this trace
    $span->setTag(\DDTrace\Tag::MANUAL_DROP, true);
  }
?>
```

{% /tab %}

{% tab title="C++" %}
Manually keep a trace:

```cpp
...
#include <datadog/tags.h>
#include <datadog/trace_segment.h>
#include <datadog/sampling_priority.h>
...

dd::SpanConfig span_cfg;
span_cfg.resource = "operation_name";

auto span = tracer.create_span(span_cfg);
// Always keep this trace
span.trace_segment().override_sampling_priority(int(dd::SamplingPriority::USER_KEEP));
//method impl follows
```

Manually drop a trace:

```cpp
...
#include <datadog/tags.h>
#include <datadog/trace_segment.h>
#include <datadog/sampling_priority.h>
...

using namespace dd = datadog::tracing;

dd::SpanConfig span_cfg;
span_cfg.resource = "operation_name";

auto another_span = tracer.create_span(span_cfg);
// Always drop this trace
span.trace_segment().override_sampling_priority(int(dd::SamplingPriority::USER_DROP));
//method impl follows
```

{% /tab %}

Set Manual Keep before context propagation. If set after context propagation, the entire trace may not be kept across services. Because this decision is set at the tracing client, the trace can still be dropped by the Agent or server based on sampling rules.

## Single spans{% #single-spans %}

`ingestion_reason: single_span`

If you need to sample a specific span but don't need the full trace, SDKs allow you to set a sampling rate for a single span.

For example, if you are building [metrics from spans](https://docs.datadoghq.com/tracing/trace_pipeline/generate_metrics.md) to monitor specific services, you can configure span sampling rules so that these metrics are based on 100% of application traffic, without ingesting 100% of traces for all requests flowing through the service.

This feature is available for Datadog Agent v[7.40.0](https://github.com/DataDog/datadog-agent/releases/tag/7.40.0)+.

**Note**: Single span sampling rules **cannot** be used to drop spans that are kept by head-based sampling, only to keep additional spans that are dropped by head-based sampling.

{% tab title="Java" %}
Starting in SDK [version 1.7.0](https://github.com/DataDog/dd-trace-java/releases/tag/v1.7.0), for Java applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect 100% of the spans from the service named `my-service`, for the operation `http.request`, up to 50 spans per second:

```
@env DD_SPAN_SAMPLING_RULES=[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]
```

Read more about sampling controls in the [Java SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/java.md).
{% /tab %}

{% tab title="Python" %}
Starting from version [v1.4.0](https://github.com/DataDog/dd-trace-py/releases/tag/v1.4.0), for Python applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service named `my-service`, for the operation `http.request`, up to `50` spans per second:

```
@env DD_SPAN_SAMPLING_RULES=[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]
```

Read more about sampling controls in the [Python SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/python.md).
{% /tab %}

{% tab title="Ruby" %}
Starting from version [v1.5.0](https://github.com/DataDog/dd-trace-rb/releases/tag/v1.5.0), for Ruby applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service named `my-service`, for the operation `http.request`, up to `50` spans per second:

```
@env DD_SPAN_SAMPLING_RULES=[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]
```

Read more about sampling controls in the [Ruby SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/ruby.md#sampling).
{% /tab %}

{% tab title="Go" %}
Starting from version [v1.41.0](https://github.com/DataDog/dd-trace-go/releases/tag/v1.41.0), for Go applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service named `my-service`, for the operation `http.request`, up to `50` spans per second:

```
@env DD_SPAN_SAMPLING_RULES=[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]
```

Starting from version [v1.60.0](https://github.com/DataDog/dd-trace-go/releases/tag/v1.60.0), for Go applications, set by-resource and by-tags **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service for the resource `POST /api/create_issue`, for the tag `priority` with value `high`:

```
@env DD_SPAN_SAMPLING_RULES=[{"resource": "POST /api/create_issue", "tags": { "priority":"high" }, "sample_rate":1.0}]
```

Read more about sampling controls in the [Go SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/go.md).
{% /tab %}

{% tab title="Node.js" %}
For Node.js applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service named `my-service`, for the operation `http.request`, up to `50` spans per second:

```
@env DD_SPAN_SAMPLING_RULES=[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]
```

Read more about sampling controls in the [Node.js SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/nodejs.md).
{% /tab %}

{% tab title="PHP" %}
Starting from version [v0.77.0](https://github.com/DataDog/dd-trace-php/releases/tag/0.77.0), for PHP applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service named `my-service`, for the operation `http.request`, up to `50` spans per second:

```
@env DD_SPAN_SAMPLING_RULES=[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]
```

Read more about sampling controls in the [PHP SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/php.md).
{% /tab %}

{% tab title="C++" %}
Starting from version [v0.1.0](https://github.com/DataDog/dd-trace-cpp/releases/tag/v0.1.0), for C++ applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service named `my-service`, for the operation `http.request`, up to `50` spans per second:

```
@env DD_SPAN_SAMPLING_RULES=[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]
```

{% /tab %}

{% tab title=".NET" %}
Starting from version [v2.18.0](https://github.com/DataDog/dd-trace-dotnet/releases/tag/v2.18.0), for .NET applications, set by-service and by-operation name **span** sampling rules with the `DD_SPAN_SAMPLING_RULES` environment variable.

For example, to collect `100%` of the spans from the service named `my-service`, for the operation `http.request`, up to `50` spans per second:

```
#using powershell
$env:DD_SPAN_SAMPLING_RULES='[{"service": "my-service", "name": "http.request", "sample_rate":1.0, "max_per_second": 50}]'

#using JSON file   
{
    "DD_SPAN_SAMPLING_RULES": "[{\"service\": \"my-service\", \"name\": \"http.request\", \"sample_rate\": 1.0, \"max_per_second\": 50}]"
}
```

Read more about sampling controls in the [.NET SDK documentation](https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/dotnet-core.md).
{% /tab %}

{% alert level="warning" %}
The legacy [App Analytics](https://docs.datadoghq.com/tracing/legacy_app_analytics.md) mechanism is fully deprecated. Use **Single span sampling** (described above) to ingest individual spans, or head-based sampling to ingest complete traces.
{% /alert %}

## Product ingested spans{% #product-ingested-spans %}

### RUM traces{% #rum-traces %}

`ingestion_reason:rum`

A request from a web or mobile application generates a trace when the backend services are instrumented. [The APM integration with Real User Monitoring](https://docs.datadoghq.com/real_user_monitoring/correlate_with_other_telemetry/apm.md) links web and mobile application requests to their corresponding backend traces so you can see your full frontend and backend data through one lens.

Beginning with version `4.30.0` of the RUM browser SDK, you can control ingested volumes and keep a sampling of the backend traces by configuring the `traceSampleRate` initialization parameter. Set `traceSampleRate` to a number between `0` and `100`. If no `traceSampleRate` value is set, a default of 100% of the traces coming from the browser requests are sent to Datadog.

You can also control the trace sampling rate in other SDKs:

| SDK          | Parameter             | Minimum version                                                                                                                                                                                                     |
| ------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Browser      | `traceSampleRate`     | [v4.30.0](https://github.com/DataDog/browser-sdk/releases/tag/v4.30.0)                                                                                                                                              |
| iOS          | `tracingSamplingRate` | [1.11.0](https://github.com/DataDog/dd-sdk-ios/releases/tag/1.11.0) *Sampling rate is reported in the Ingestion Control Page since [1.13.0](https://github.com/DataDog/dd-sdk-ios/releases/tag/1.13.0)*             |
| Android      | `traceSampleRate`     | [1.13.0](https://github.com/DataDog/dd-sdk-android/releases/tag/1.13.0) *Sampling rate is reported in the Ingestion Control Page since [1.15.0](https://github.com/DataDog/dd-sdk-android/releases/tag/1.15.0)*     |
| Flutter      | `tracingSamplingRate` | [1.0.0](https://github.com/DataDog/dd-sdk-flutter/releases/tag/datadog_flutter_plugin%2Fv1.0.0)                                                                                                                     |
| React Native | `tracingSamplingRate` | [1.0.0](https://github.com/DataDog/dd-sdk-reactnative/releases/tag/1.0.0) *Sampling rate is reported in the Ingestion Control Page since [1.2.0](https://github.com/DataDog/dd-sdk-reactnative/releases/tag/1.2.0)* |

### Synthetic traces{% #synthetic-traces %}

`ingestion_reason:synthetics` and `ingestion_reason:synthetics-browser`

HTTP and browser tests generate traces when the backend services are instrumented. [The APM integration with Synthetic Testing](https://docs.datadoghq.com/synthetics/apm.md) links your synthetic tests with the corresponding backend traces. Navigate from a test run that failed to the root cause of the issue by looking at the trace generated by that test run.

By default, 100% of synthetic HTTP and browser tests generate backend traces.

### Other products{% #other-products %}

Some additional ingestion reasons are attributed to spans that are generated by specific Datadog products:

| Product                             | Ingestion Reason    | Ingestion Mechanism Description                                                                                                                                              |
| ----------------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Serverless                          | `lambda` and `xray` | Your traces received from the [Serverless applications](https://docs.datadoghq.com/serverless/distributed_tracing.md) traced with Datadog SDKs or the AWS X-Ray integration. |
| App and API Protection              | `appsec`            | Traces ingested from Datadog SDKs and flagged by [AAP](https://docs.datadoghq.com/security/application_security.md) as a threat.                                             |
| Data Observability: Jobs Monitoring | `data_jobs`         | Traces ingested from the Datadog Java Tracer Spark integration or the Databricks integration.                                                                                |

## Ingestion mechanisms in OpenTelemetry{% #ingestion-mechanisms-in-opentelemetry %}

`ingestion_reason:otel`

Depending on your setup with the OpenTelemetry SDKs (using the OpenTelemetry Collector or the Datadog Agent), you have multiple ways of controlling ingestion sampling. See [Ingestion Sampling with OpenTelemetry](https://docs.datadoghq.com/opentelemetry/guide/ingestion_sampling_with_opentelemetry.md) for details about the options available for sampling at the OpenTelemetry SDK, OpenTelemetry Collector, and Datadog Agent level in various OpenTelemetry setups.

## Further Reading{% #further-reading %}

- [Ingestion Controls](https://docs.datadoghq.com/tracing/trace_pipeline/ingestion_controls.md)
- [Trace Retention](https://docs.datadoghq.com/tracing/trace_pipeline/trace_retention.md)
- [Usage Metrics](https://docs.datadoghq.com/tracing/trace_pipeline/metrics.md)
- [Optimizing Datadog at scale: Cost-efficient observability at Zendesk](https://www.datadoghq.com/blog/zendesk-cost-optimization/#improving-tracing-efficiency-through-targeted-changes)
- [APM Rate Limiting and Retention](https://learn.datadoghq.com/courses/apm-rate-limit-retention)