---
title: Track Feature Flag Changes
description: >-
  Learn how to track, correlate, and remediate feature flag changes in Datadog
  to identify and resolve issues caused by flag updates.
breadcrumbs: Docs > Change Tracking > Track Feature Flag Changes
---

# Track Feature Flag Changes

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). ().
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Feature flag tracking in Datadog helps you correlate flag changes with system performance issues, accelerating incident resolution. By tracking when flags are toggled, updated, or rolled out, you can identify whether a feature flag change caused a performance degradation or outage.

With feature flag tracking, you can:

- View flag changes directly in your dashboards, monitors, and service pages
- Automatically identify which services are affected by a flag change
- Roll back problematic flags directly from inside Datadog

{% image
   source="https://datadog-docs.imgix.net/images/change_tracking/monitor.c4161703dc9819550264b83fe89541d1.png?auto=format"
   alt="The monitor graph shows the error spike correlated with the flag change." /%}

## Track feature flags{% #track-feature-flags %}

Datadog supports tracking LaunchDarkly flags using the [LaunchDarkly integration](https://docs.datadoghq.com/integrations/launchdarkly/#setup) or tracking feature flags from other feature flag providers using the [Events API](https://docs.datadoghq.com/api/latest/events/).

### LaunchDarkly flags{% #launchdarkly-flags %}

To track LaunchDarkly feature flags in your services' Change Tracking timeline:

1. Enable the [Datadog integration](https://docs.datadoghq.com/integrations/launchdarkly/#setup) in LaunchDarkly.
1. Go to **Flags > `<your-feature-flag-name>` in LaunchDarkly.
1. In **Datadog tags**, add a tag with key `service` and value `<your-service-name>`, matching your Datadog service name exactly.
1. Click **Save changes**.

For example, to link a flag to the `payments_api` service used in the examples below, you would set the tag value to `payments_api`. After you submit the event, you can navigate to the [Software Catalog](https://app.datadoghq.com/software), select the `payments_api` service, and see the `fallback_payments_test` feature flag event in the Change Tracking timeline.

### Custom feature flags{% #custom-feature-flags %}

Send feature flag events from any provider using the [Events API](https://docs.datadoghq.com/api/latest/events/). Create a `change` category event and include a service tag to link the event to your service.

#### Recommended tags{% #recommended-tags %}

When sending custom feature flag change events, include the following fields to enable accurate filtering and cross-product correlation within Datadog:

- **impacted\_resources** (with type `service`): Add the relevant service name to the `impacted_resources` array to associate the feature flag change with the affected service.
- **env tag**: Specify the environment where the change occurred (for example, production, staging, or development).

If these tags cannot be added at event creation time, see the next section for guidance on automatic enrichment.

Example request:

```json
{
  "data": {
    "attributes": {
      "aggregation_key": "string",
      "attributes": {
        "author": {
          "name": "datadog@datadog.com",
          "type": "user"
        },
        "change_metadata": {
          "dd": {
            "team": "datadog_team",
            "user_email": "datadog@datadog.com",
            "user_id": "datadog_user_id",
            "user_name": "datadog_username"
          },
          "resource_link": "datadog.com/feature/fallback_payments_test"
        },
        "changed_resource": {
          "name": "fallback_payments_test",
          "type": "feature_flag"
        },
        "impacted_resources": [
          {
            "name": "payments_api",
            "type": "service"
          }
        ],
        "new_value": {
          "enabled": true,
          "percentage": "50%",
          "rule": {
            "datacenter": "devcycle.us1.prod"
          }
        },
        "prev_value": {
          "enabled": true,
          "percentage": "10%",
          "rule": {
            "datacenter": "devcycle.us1.prod"
          }
        }
      },
      "category": "change",
      "message": "payment_processed feature flag has been enabled",
      "tags": [
        "env:test"
      ],
      "timestamp": "string",
      "title": "payment_processed feature flag updated"
    },
    "type": "event"
  }
}
```

## Automatically detect affected services{% #automatically-detect-affected-services %}

In addition to tracking feature flag configuration changes through the LaunchDarkly integration or the Events API, Datadog can automatically detect which services evaluate a flag by using APM traces or metrics. This provides real-time visibility into flag usage across your system, especially when the same flag is evaluated by multiple services.

### Trace-based enrichment{% #trace-based-enrichment %}

Trace-based enrichment uses APM traces to automatically associate feature flag changes with the relevant Datadog services. The following section details how to implement it in your codebase.

#### Setup{% #setup %}

To automatically detect services using a feature flag, instrument your feature flag evaluation code with the APM tracing library. This allows Datadog to automatically detect all services that evaluate a specific flag, even if they weren't originally tagged.

1. [Instrument your feature flag evaluation code](https://docs.datadoghq.com/tracing/trace_collection/) using the Datadog tracing library.
1. Create a custom span with the operation name `experiments.IsEnabled` to track feature flag evaluations.
1. Tag the span with `experiment_id:<flag-id>`, where `<flag-id>` matches the feature flag ID.

For example:

```python
# Trace feature flag evaluation to enable auto-detection
with tracer.trace("experiments.IsEnabled") as span:
    span.set_tag("experiment_id", "fallback_payments_test") # adds the flag name as a span tag
    # Your existing feature flag evaluation code
    flag_value = evaluate_flag("fallback_payments_test")
    span.set_tag("experiment_value", flag_value) # adds the evaluated flag value as a span tag
```

### Metrics-based enrichment{% #metrics-based-enrichment %}

Metric-based enrichment uses Datadog metrics to enrich your feature flag changes with service context. The following section explains how to implement it in your codebase.

#### Setup{% #setup-1 %}

Send the `dd.dynamic_config.is_experiment_enabled` metric with your experiment status.

```python
from datadog import initialize, statsd

initialize(statsd_host='localhost', statsd_port=8125)

# Any metric type is supported
statsd.gauge(
    'dd.dynamic_config.is_experiment_enabled',
    1,  # 1 if enabled, 0 if disabled
    tags=[
        'experiment.id:fallback_payments_test',
        'service:payments_api'
    ]
)
```

## Remediate feature flag changes with Workflow Automation{% #remediate-feature-flag-changes-with-workflow-automation %}

When you identify that a feature flag change caused an issue, you can immediately toggle its state without leaving Datadog. This feature uses [Workflow Automation](https://app.datadoghq.com/actions/connections) to toggle LaunchDarkly flags directly from Change Tracking timelines.

{% alert level="info" %}
This feature requires Workflow Automation. See [Workflow Automation pricing](https://www.datadoghq.com/pricing/?product=workflow-automation#products).
{% /alert %}

### Setup{% #setup-2 %}

To set up feature flag toggles using Workflow Automation:

1. Go to [**Actions > Action Catalog > Connections**](https://app.datadoghq.com/actions/connections).
1. Click **New Connection**.
1. Choose *LaunchDarkly*.
1. Complete the required information, then click **Next, Confirm Access**.
1. Set access permissions for the connection.
1. Click **Create**.

### Use feature flag toggles{% #use-feature-flag-toggles %}

To toggle feature flags on or off from inside Datadog:

1. Click a LaunchDarkly feature flag change in the Change Tracking timeline.
1. Click the **Toggle Feature Flag** button.
1. Click **Run Action** to run the workflow and toggle the feature flag on or off.

{% image
   source="https://datadog-docs.imgix.net/images/change_tracking/toggle.3b48218d0ae3e35aa50cc72bc3da5fa2.png?auto=format"
   alt="The details panel for a LaunchDarkly feature flag event, showing the 'Toggle Feature Flag' button." /%}

## Further reading{% #further-reading %}

- [Change Tracking](https://docs.datadoghq.com/change_tracking/)
- [LaunchDarkly](https://docs.datadoghq.com/integrations/launchdarkly/#feature-flag-tracking-integration/)