---
title: Set Up APM Trace Enrichment for Feature Flags
description: >-
  Automatically attach feature flag evaluation data to APM traces so you can
  inspect and filter traces by flag variant.
breadcrumbs: >-
  Docs > Feature Flags > Feature Flags Guides > Set Up APM Trace Enrichment for
  Feature Flags
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Set Up APM Trace Enrichment for Feature Flags

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

APM trace enrichment automatically attaches feature flag evaluation data to your APM traces. When a feature flag is evaluated during a traced request, the SDK records which flags were evaluated and which variants were returned. This data is written to the root span and processed server-side so you can:

- **Filter traces by flag variant** in [Trace Explorer](https://docs.datadoghq.com/tracing/trace_explorer.md) using `@feature_flags.<flag_key>:<variant>` facets.
- **Debug flag-related issues** by seeing which flags were active when an error occurred.

{% alert level="warning" %}
APM trace enrichment is experimental and may change in a future release.
{% /alert %}

APM trace enrichment is available in the following SDKs:

| Language | Minimum version |
| -------- | --------------- |
| Go       | 2.8.0           |
| Java     | 1.64.1          |
| Node.js  | 5.105.0         |

## Prerequisites{% #prerequisites %}

Before setting up APM trace enrichment, confirm the following:

- Server-side feature flags are already configured and flags are evaluating in your application.
- [APM tracing](https://docs.datadoghq.com/tracing.md) is enabled and traces are flowing to Datadog.

## How APM trace enrichment works{% #how-apm-trace-enrichment-works %}

When APM trace enrichment is enabled, the Datadog OpenFeature provider hooks into the evaluation life cycle:

1. Each time a flag is evaluated, the SDK captures the evaluation metadata (flag serial ID, targeting key, and default fallback value).
1. The metadata is accumulated on the root span of the current trace.
1. When the root span finishes, the SDK writes the accumulated data as compact span tags (`ffe_flags_enc`, `ffe_subjects_enc`, `ffe_runtime_defaults`).
1. The Datadog backend decodes these tags and writes human-readable `@feature_flags.<flag_key>` facets to the span, making them searchable in Trace Explorer.

The SDK-side tags are transport-only and are stripped server-side. The tags visible to you in Trace Explorer are the decoded `@feature_flags.<flag_key>` facets.

## Enable APM trace enrichment{% #enable-apm-trace-enrichment %}

Set the following environment variable to enable span enrichment:

```bash
DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true
```

The enrichment environment variable is supported by all supported server-side SDKs. No code changes are required. Enabling the variable activates the enrichment hook automatically when the Datadog OpenFeature provider initializes. Node.js additionally supports code-level configuration as shown in the language tabs below.

### Language-specific configuration{% #language-specific-configuration %}

{% tab title="Go" %}
No additional code configuration is needed. The `DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED` environment variable enables span enrichment when the `DatadogProvider` initializes.

In the `main.go` file:

```go
package main

import (
    "log"

    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
    ddopenfeature "github.com/DataDog/dd-trace-go/v2/openfeature"
    "github.com/open-feature/go-sdk/openfeature"
)

func main() {
    tracer.Start()
    defer tracer.Stop()

    provider, err := ddopenfeature.NewDatadogProvider(ddopenfeature.ProviderConfig{})
    if err != nil {
        log.Fatalf("Failed to create provider: %v", err)
    }
    if ddProvider, ok := provider.(*ddopenfeature.DatadogProvider); ok {
        defer ddProvider.Shutdown()
    }

    if err := openfeature.SetProviderAndWait(provider); err != nil {
        log.Fatalf("Failed to set provider: %v", err)
    }

    client := openfeature.NewClient("my-service")
    // Flag evaluations now enrich APM spans automatically
}
```

{% /tab %}

{% tab title="Java" %}
No additional code configuration is needed. The `DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED` environment variable enables span enrichment. Java also supports the system property `-Ddd.experimental.flagging.provider.span.enrichment.enabled=true` as an alternative.

In the `Main.java` file:

```java
import dev.openfeature.sdk.OpenFeatureAPI;
import dev.openfeature.sdk.Client;
import datadog.trace.api.openfeature.Provider;

OpenFeatureAPI api = OpenFeatureAPI.getInstance();
api.setProviderAndWait(new Provider());
Client client = api.getClient("my-app");
// Flag evaluations now enrich APM spans automatically
```

{% /tab %}

{% tab title="Node.js" %}
You can also enable span enrichment in code:

In the `app.js` file:

```javascript
import tracer from 'dd-trace';

tracer.init({
  experimental: {
    flaggingProvider: {
      enabled: true,
      spanEnrichment: {
        enabled: true,
      },
    },
  },
});
```

{% /tab %}

## Verify APM trace enrichment{% #verify-apm-trace-enrichment %}

After deploying with span enrichment enabled:

1. Trigger requests in your application that evaluate feature flags.
1. Go to [Trace Explorer](https://docs.datadoghq.com/tracing/trace_explorer.md) and search for a recent trace from your service.
1. Open a trace and look for `@feature_flags.<flag_key>` attributes on the root span.

The SDK writes compact encoded tags (`ffe_flags_enc`, `ffe_subjects_enc`, `ffe_runtime_defaults`) to the root span. The Datadog backend decodes these and produces human-readable `@feature_flags.<flag_key>` facets. This processing takes a few seconds after the span is ingested.

After backend processing, the root span contains attributes such as the following examples:

| Example attribute              | Example value |
| ------------------------------ | ------------- |
| `@feature_flags.checkout-flow` | `treatment`   |
| `@feature_flags.dark-mode`     | `control`     |

Each attribute key is `@feature_flags.<flag_key>` and the value is the variant returned by the evaluation.

### Troubleshooting{% #troubleshooting %}

If `@feature_flags.<flag_key>` attributes do not appear on your traces:

- Confirm span enrichment is enabled (`DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true`).
- Verify that your application is evaluating flags during traced requests. Enrichment only occurs when a flag is evaluated while a trace is active.
- Wait a few seconds after the span is ingested. The `@feature_flags.<flag_key>` facets are derived by backend processing and do not appear in raw span metadata.
- For debugging, inspect raw span metadata for the `ffe_flags_enc` tag. If this tag is present, the SDK is emitting enrichment data. Either the backend has not processed it yet, or the feature flag gate is not enabled for your organization.
- If flags are not evaluating at all, see [Server-Side Feature Flags](https://docs.datadoghq.com/feature_flags/server.md) for setup and language-specific troubleshooting.

## Search and filter by flag variant{% #search-and-filter-by-flag-variant %}

These examples use the `@feature_flags.<flag_key>` facets to filter traces in Trace Explorer:

| Use case                             | Example query                                                              |
| ------------------------------------ | -------------------------------------------------------------------------- |
| Traces for a specific variant        | `@feature_flags.checkout-flow:treatment`                                   |
| Errors under a variant               | `@feature_flags.checkout-flow:treatment status:error`                      |
| Any trace where a flag was evaluated | `@feature_flags.checkout-flow:*`                                           |
| Multiple flags on the same request   | `@feature_flags.checkout-flow:treatment @feature_flags.new-search:enabled` |
| Scoped to service and environment    | `env:production service:api-gateway @feature_flags.rate-limit-v2:enabled`  |

## Use enriched traces across Datadog{% #use-enriched-traces-across-datadog %}

Feature flag attributes on traces are available across Datadog:

- **Monitors**: Alert when the error count for a specific variant exceeds a threshold to catch variant-specific regressions.
- **Dashboards**: Add a timeseries widget comparing p99 latency across variants using `@feature_flags.<flag_key>` as a group-by dimension.
- **Notebooks**: Build an investigation notebook comparing performance across feature flag variants.
- **Visualizations**: Use the Top List view in Trace Explorer to verify that rollout traffic distribution matches your targeting rules.

## Limits{% #limits %}

The SDK enforces the following per-span limits to bound payload size:

| Limit                         | Value                      |
| ----------------------------- | -------------------------- |
| Flag serial IDs per span      | 128 to 200 (varies by SDK) |
| Subjects per span             | 10 to 25 (varies by SDK)   |
| Runtime default keys per span | 5                          |
| Runtime default value length  | 64 characters (truncated)  |

Evaluations beyond these limits are dropped for that span.

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Server-Side Feature Flags](https://docs.datadoghq.com/feature_flags/server.md)
- [Set Up Server-Side Flag Evaluation Metrics](https://docs.datadoghq.com/feature_flags/guide/server_flag_evaluation_metrics.md)
- [Trace Explorer](https://docs.datadoghq.com/tracing/trace_explorer.md)