Overview

APM trace enrichment automatically attaches feature flag evaluation data to your APM traces. When a feature flag is evaluated during a traced request, the SDK records which flags were evaluated and which variants were returned. This data is written to the root span and processed server-side so you can:

• Filter traces by flag variant in Trace Explorer using @feature_flags.<flag_key>:<variant> facets.
• Debug flag-related issues by seeing which flags were active when an error occurred.

APM trace enrichment is available in the following SDKs:

Prerequisites

Before setting up APM trace enrichment, confirm the following:

• Server-side feature flags are already configured and flags are evaluating in your application.
• APM tracing is enabled and traces are flowing to Datadog.

How APM trace enrichment works

When APM trace enrichment is enabled, the Datadog OpenFeature provider hooks into the evaluation life cycle:

1. Each time a flag is evaluated, the SDK captures the evaluation metadata (flag serial ID, targeting key, and default fallback value).
2. The metadata is accumulated on the root span of the current trace.
3. When the root span finishes, the SDK writes the accumulated data as compact span tags (ffe_flags_enc, ffe_subjects_enc, ffe_runtime_defaults).
4. The Datadog backend decodes these tags and writes human-readable @feature_flags.<flag_key> facets to the span, making them searchable in Trace Explorer.

The SDK-side tags are transport-only and are stripped server-side. The tags visible to you in Trace Explorer are the decoded @feature_flags.<flag_key> facets.

Enable APM trace enrichment

Set the following environment variable to enable span enrichment:

DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true

The enrichment environment variable is supported by all supported server-side SDKs. No code changes are required. Enabling the variable activates the enrichment hook automatically when the Datadog OpenFeature provider initializes. Node.js additionally supports code-level configuration as shown in the language tabs below.

Language-specific configuration

package main

import (
    "log"

    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
    ddopenfeature "github.com/DataDog/dd-trace-go/v2/openfeature"
    "github.com/open-feature/go-sdk/openfeature"
)

func main() {
    tracer.Start()
    defer tracer.Stop()

    provider, err := ddopenfeature.NewDatadogProvider(ddopenfeature.ProviderConfig{})
    if err != nil {
        log.Fatalf("Failed to create provider: %v", err)
    }
    if ddProvider, ok := provider.(*ddopenfeature.DatadogProvider); ok {
        defer ddProvider.Shutdown()
    }

    if err := openfeature.SetProviderAndWait(provider); err != nil {
        log.Fatalf("Failed to set provider: %v", err)
    }

    client := openfeature.NewClient("my-service")
    // Flag evaluations now enrich APM spans automatically
}

import dev.openfeature.sdk.OpenFeatureAPI;
import dev.openfeature.sdk.Client;
import datadog.trace.api.openfeature.Provider;

OpenFeatureAPI api = OpenFeatureAPI.getInstance();
api.setProviderAndWait(new Provider());
Client client = api.getClient("my-app");
// Flag evaluations now enrich APM spans automatically

import tracer from 'dd-trace';

tracer.init({
  experimental: {
    flaggingProvider: {
      enabled: true,
      spanEnrichment: {
        enabled: true,
      },
    },
  },
});

Verify APM trace enrichment

After deploying with span enrichment enabled:

1. Trigger requests in your application that evaluate feature flags.
2. Go to Trace Explorer and search for a recent trace from your service.
3. Open a trace and look for @feature_flags.<flag_key> attributes on the root span.

The SDK writes compact encoded tags (ffe_flags_enc, ffe_subjects_enc, ffe_runtime_defaults) to the root span. The Datadog backend decodes these and produces human-readable @feature_flags.<flag_key> facets. This processing takes a few seconds after the span is ingested.

After backend processing, the root span contains attributes such as the following examples:

Each attribute key is @feature_flags.<flag_key> and the value is the variant returned by the evaluation.

Troubleshooting

If @feature_flags.<flag_key> attributes do not appear on your traces:

• Confirm span enrichment is enabled (DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true).
• Verify that your application is evaluating flags during traced requests. Enrichment only occurs when a flag is evaluated while a trace is active.
• Wait a few seconds after the span is ingested. The @feature_flags.<flag_key> facets are derived by backend processing and do not appear in raw span metadata.
• For debugging, inspect raw span metadata for the ffe_flags_enc tag. If this tag is present, the SDK is emitting enrichment data. Either the backend has not processed it yet, or the feature flag gate is not enabled for your organization.
• If flags are not evaluating at all, see Server-Side Feature Flags for setup and language-specific troubleshooting.

Search and filter by flag variant

These examples use the @feature_flags.<flag_key> facets to filter traces in Trace Explorer:

Use enriched traces across Datadog

Feature flag attributes on traces are available across Datadog:

• Monitors: Alert when the error count for a specific variant exceeds a threshold to catch variant-specific regressions.
• Dashboards: Add a timeseries widget comparing p99 latency across variants using @feature_flags.<flag_key> as a group-by dimension.
• Notebooks: Build an investigation notebook comparing performance across feature flag variants.
• Visualizations: Use the Top List view in Trace Explorer to verify that rollout traffic distribution matches your targeting rules.

Limits

The SDK enforces the following per-span limits to bound payload size:

Evaluations beyond these limits are dropped for that span.

Further reading

Additional helpful documentation, links, and articles:

Server-Side Feature FlagsDOCUMENTATION Set Up Server-Side Flag Evaluation MetricsGUIDE Trace ExplorerDOCUMENTATION