---
title: Create Experiment Metrics
description: Create the metrics you want to measure in your experiments.
breadcrumbs: Docs > Experiments > Create Experiment Metrics
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Create Experiment Metrics

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Create the metrics you want to measure in your experiments. You can use data from Real User Monitoring (RUM), Product Analytics, Agent Observability, or your own warehouse to create Datadog Experiments metrics.

{% alert level="info" %}
If your organization uses custom roles, you must have the appropriate [Product Analytics permissions](https://docs.datadoghq.com/account_management/rbac/permissions.md#product-analytics) to create experiment metrics.
{% /alert %}

## Create a metric{% #create-a-metric %}

Select your data source:

{% tab title="Product Analytics or RUM" %}
### Prerequisites{% #prerequisites %}

To create a metric from Product Analytics or RUM data, you must have Datadog's [client-side SDK](https://docs.datadoghq.com/real_user_monitoring.md#get-started) installed in your application and be actively capturing data. If you have not yet configured your SDK, select your application type to get started:

- [Android and Android TV](https://docs.datadoghq.com/real_user_monitoring/application_monitoring/android/setup.md?tab=kotlin)
- [iOS and tvOS](https://docs.datadoghq.com/real_user_monitoring/application_monitoring/ios/setup.md?tab=swift-package-manager--spm)
- [Browser (JavaScript)](https://docs.datadoghq.com/real_user_monitoring/application_monitoring/browser/setup/client.md?tab=npm)
- [React Native](https://docs.datadoghq.com/real_user_monitoring/application_monitoring/react_native/setup.md?platform=react_native)

Product Analytics uses the same SDKs and configuration as Real User Monitoring (RUM). After you configure your SDK using the RUM setup documentation, create your metric in the Product Analytics UI.

### Create a metric using Product Analytics or RUM data{% #create-a-metric-using-product-analytics-or-rum-data %}

To create a metric for your experiment:

1. Navigate to the [Metrics page](https://app.datadoghq.com/product-analytics/experimentation-metrics) in Datadog Product Analytics.
1. Select the Metrics tab and click Create Metric at the top right corner.
1. Add a Metric name and, optionally, a Description.
1. Under the Metric definition section, click Select an event to open the event picker. The chart on the right updates in real time as you configure your metric.
   1. Search for a specific event, or use the By Type filter to browse by event type.
1. Select an aggregation method from the dropdown. The default is Count of events.
1. Click Add Filter to filter your metric by additional properties.
1. (Optional) Under the Additional settings section:
   1. Toggle on Mark as certified to indicate that this metric is approved for important decision-making. This requires the Product Analytics Certified Metrics Write permission.
   1. Adjust the Experiment settings and Units as needed. The defaults work for most use cases.
1. Click Save.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_create_new_metric.9077f7da99d40ac5aee04415d626f361.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_create_new_metric.9077f7da99d40ac5aee04415d626f361.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create Metric page with the Metric name set to 'Example metric', the 'click on ADD TO CART' event selected, the aggregation method dropdown set to Count of events, the Additional settings section, a bar chart preview on the right, and the Save button highlighted." /%}

### Add filters{% #add-filters %}

You can filter your metric by selecting an Event properties filter, such as Service, Country, or Device Type. Use the By Data Type filter to narrow the list of available properties by type (for example, String or Boolean).

If you do not see the property you need, type the property name in the Custom property field (for example, `@context.tracking`) and click Add.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_filter_by_2.8b2a00b4561f4c9f1dfdf4ee041ec43a.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_filter_by_2.8b2a00b4561f4c9f1dfdf4ee041ec43a.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Filter by panel open within the Metric definition section, showing All Properties selected, Event properties such as Application Id, Service, Browser Name, and Country in the center, a By Data Type filter with Numerical, String, and Boolean options on the left, and a Custom property section at the bottom with a text field showing the placeholder 'e.g. @context.tracking' and an Add button." /%}

{% /tab %}

{% tab title="Agent Observability" %}

{% alert level="info" %}
**Note**: Online experiments for Agent Observability are in Preview. Contact your Datadog representative to request access.
{% /alert %}

### Prerequisites{% #prerequisites %}

Before you create an experiment metric from Agent Observability data:

- [Instrument your LLM application with Agent Observability](https://docs.datadoghq.com/llm_observability/instrument.md) and send traces to Datadog.
- To use cost or token data, make sure your traces include token usage. Datadog uses token usage, model, and provider information to calculate [estimated costs](https://docs.datadoghq.com/llm_observability/investigate/cost.md).
- To use an evaluation, submit or configure a custom [Agent Observability evaluation](https://docs.datadoghq.com/llm_observability/investigate/evaluations.md) with a numeric `score` value.
- Add a `subject_identifier` tag to each trace or evaluation. The value must match the `targetingKey` that you use to evaluate the experiment's feature flag. For setup instructions, see [Run an Online Experiment on an LLM Application](https://docs.datadoghq.com/experiments/guide/run_online_experiment_on_llm_application.md).

{% alert level="warning" %}
**Supported evaluation type**: Experiment metrics support only Agent Observability evaluations with a `score` metric type. Boolean, categorical, and other evaluation types are not available in the metric picker.
{% /alert %}

### Create a metric from Agent Observability data{% #create-a-metric-from-agent-observability-data %}

To create the metric:

1. Navigate to the [Metrics page](https://app.datadoghq.com/product-analytics/experimentation-metrics) in Datadog Product Analytics.
1. Select the Metrics tab and click Create Metric.
1. Add a Metric name and, optionally, a Description.
1. In the Metric definition section, click Select an event.
1. Choose one of the following Agent Observability sources.

#### Agent Spans{% #agent-spans %}

Select the Agent Spans tab, then select one of the following:

- Total estimated cost: Average estimated cost across completed Agent Observability traces.
- Total tokens: Average total token usage across completed Agent Observability traces.

The cost and token templates default to Average of their corresponding trace property. You can change the aggregation, select another property or custom property path, add filters, or create a ratio.

{% alert level="info" %}
Agent Observability estimated cost values use nanodollars (1 nanodollar = 10⁻⁹ USD). For details about cost calculations and supported models, see [Cost](https://docs.datadoghq.com/llm_observability/investigate/cost.md).
{% /alert %}

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_agent_observability_agent_spans.c6d5b94142bdea180c9428b0ec94b483.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_agent_observability_agent_spans.c6d5b94142bdea180c9428b0ec94b483.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create Metric event picker with Agent Spans selected, showing All agent span events, Total estimated cost, and Total tokens, with Total estimated cost selected and its description displayed." /%}

#### Evaluations{% #evaluations %}

Select the Evaluations tab, then:

1. Search for and select the custom score evaluation you want to measure.
1. Review the evaluation details, such as its ML application, scope, prompt, and model.
1. Configure the aggregation and any filters. Evaluation metrics default to the average score.

Before you create the experiment metric, run the evaluation on at least one span. An evaluation appears in the picker only after it has run.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_agent_observability_evaluations.692aa8d3657af85d4b72febbd16d466e.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_agent_observability_evaluations.692aa8d3657af85d4b72febbd16d466e.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create Metric event picker with Evaluations selected, showing the faithfulness, user_score, and verbosity score evaluations, with faithfulness selected." /%}

#### Save the metric{% #save-the-metric %}

After you configure either source:

1. (Optional) Under Additional settings, mark the metric as certified, adjust its experiment settings, or configure its units.
1. Click Save.

{% /tab %}

{% tab title="Warehouse" %}
### Prerequisites{% #prerequisites %}

To create a metric from your warehouse data, you must [connect your warehouse to Datadog](https://docs.datadoghq.com/experiments/guide/connecting_a_data_warehouse.md). Datadog supports BigQuery, Databricks, Redshift, and Snowflake.

After you connect your warehouse, create a SQL Model to map your data to Datadog, then use the model to create a metric.

### Create a SQL Model{% #create-a-sql-model %}

Write your SQL query to define and preview your data, then configure your model to map the data to Datadog.

#### Write your SQL{% #write-your-sql %}

Start by writing a query to retrieve your data:

1. Navigate to the [Metrics page](https://app.datadoghq.com/product-analytics/experimentation-metrics) in Datadog Product Analytics.
1. Select the Metric SQL Models tab and click Create SQL Model.
1. In the Write SQL section, enter a SQL query that returns your data of interest. The SQL editor supports `SELECT * FROM` and more advanced SQL statements.
1. Click Run to preview your data.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_sql_models_writesql_1.23c8ad232236223a71285f8489f3dbb8.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_sql_models_writesql_1.23c8ad232236223a71285f8489f3dbb8.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Write SQL section of the Create Metric SQL Model page showing a SELECT query for user_id, revenue_timestamp, and amount from a revenue orders table, with a successful query preview below displaying USER_ID, REVENUE_TIMESTAMP, and AMOUNT columns." /%}

For large tables, use [SQL template variables](https://docs.datadoghq.com/experiments/concepts/sql_template_variables.md) to push Datadog's date filters into your query and reduce the amount of data your warehouse scans on each run.

#### Map your warehouse data to Datadog{% #map-your-warehouse-data-to-datadog %}

After previewing your data, map it to Datadog. In the Structure your model section:

1. Add a Metric SQL Model Name (for example, **Revenue Orders**).
1. (Optional) Toggle on Mark as certified to indicate that this SQL model is approved for important decision-making. This requires the Product Analytics Certified Metrics Write permission.
1. Map the columns in your warehouse table to the following:
   - Timestamp column
     - The column that lists the timestamp associated with the metric event.
     - The analysis only includes rows created after the subject enrolls in the experiment.
   - Subject Type
     - The attribute that Datadog uses to randomly assign experiment groups.
     - You can define the subject type and its default warehouse column on the [Subject Types](https://app.datadoghq.com/product-analytics/experiments/settings/subject-types) page. For example, you can use `user_id` for an individual user or `org_id` for an organization account.
   - Measures (optional)
     - The numeric columns from your warehouse table that Datadog can aggregate into metrics (for example, a `revenue` or `amount` column).
     - Each SQL model automatically includes an each record measure. Use this measure to count the number of relevant rows in the table for a specific experiment subject.
1. Click Create Metric SQL Model to save your SQL model.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metrics_sql_model_structure4.01380ad7ba2dc78598935ea9b6afead5.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metrics_sql_model_structure4.01380ad7ba2dc78598935ea9b6afead5.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Structure your model panel with the Metric SQL Model Name field set to 'Revenue Orders' and highlighted, a Mark as certified toggle, Timestamp column set to REVENUE_TIMESTAMP, Subject Type set to User (@usr.id) with USER_ID selected in the column selector, a Measures dropdown showing 'Revenue Orders (each record)', and the Create Metric SQL Model button highlighted." /%}

### Create a metric using your SQL model{% #create-a-metric-using-your-sql-model %}

After you create your SQL model, use it to create a metric:

1. Navigate to the [Metrics page](https://app.datadoghq.com/product-analytics/experimentation-metrics) in Datadog Product Analytics.
1. Select the Metrics tab and click Create Metric at the top right corner.
1. Add a Metric name and, optionally, a Description.
1. Under the Metric definition section, click Select an event to open the event picker. The chart on the right updates in real time as you configure your metric.
   1. Select the relevant SQL model. Your SQL models appear under their data source (for example, **Revenue Orders** under **Snowflake**).
1. Select an aggregation method from the dropdown.
1. (Optional) Under the Additional settings section:
   1. Toggle on Mark as certified to indicate this metric is approved for important decision-making. This requires the Product Analytics Certified Metrics Write permission.
   1. Adjust the Experiment settings and Units as needed. The defaults work for most use cases.
1. Click Save.

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_from_sqlmodel_2.1253be9e311ce9872bee98b9c00c085d.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_create_metric_from_sqlmodel_2.1253be9e311ce9872bee98b9c00c085d.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Create Metric event picker showing All Events selected, with event types including Snowflake, Actions, Views, Sessions, Errors, and Long Tasks on the left, and the Revenue Orders SQL model highlighted under Snowflake on the right, showing Measures: amount and Filterable dimensions: N/A." /%}

{% /tab %}

## Aggregation methods{% #aggregation-methods %}

Aggregation methods determine how Datadog summarizes data for each experiment subject. An experiment subject is the unit that Datadog randomizes for the experiment. This is typically a user, but can also be an organization, cookie, or device, depending on how you set up your experiment.

Datadog Experiments supports the following aggregation methods:

- Count of events (default)
- Count of unique users (useful for conversion metrics)
- Sum of an event property (useful for revenue metrics)
- Distinct values of an event property (useful for unique pages viewed metrics)
- Percentile of an event property (useful for latency metrics)
- Average of an event property (useful for satisfaction metrics)

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_default_metric_agg_1.54ef416d873bc80b6dab22c6c1fce63c.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_default_metric_agg_1.54ef416d873bc80b6dab22c6c1fce63c.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The aggregation method dropdown showing Count of unique users (selected) and Count of events at the top, followed by a SELECT A MEASURE section with Sum of, Distinct values of, Percentile, and Average of options, with a description reading 'The number of users who performed the event' on the right." /%}

Datadog computes metrics for each experiment subject. For example, a Count of events metric on an experiment randomized by user calculates the total number of events for all users in the variant (experiment group) divided by the number of users in that variant.

### Ratio metrics{% #ratio-metrics %}

Click Create Ratio to divide your metric by a value other than the default number of experiment subjects. The denominator can use any of the aggregation methods. For example, divide purchases by product page views to measure conversion at a specific step in the funnel, rather than across all enrolled users.

Datadog accounts for correlations between the numerator and denominator using the [delta method](https://en.wikipedia.org/wiki/Delta_method).

{% image
   source="https://docs.dd-static.net/images/product_analytics/experiment/exp_create_ratio_new_ui.8299cead08b0d934d3a9cde31257e615.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/product_analytics/experiment/exp_create_ratio_new_ui.8299cead08b0d934d3a9cde31257e615.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Metric definition section showing the 'click on ADD TO CART' event with Count of events aggregation and an Add Filter option, the Create Ratio button highlighted below, and the Additional settings section with Mark as certified toggle, Experiment settings, and Units." /%}

## Advanced options{% #advanced-options %}

Datadog Experiments supports the following advanced options. These can be modified under Additional settings > Experiment settings when creating a metric.

{% dl %}

{% dt %}
Time frame filters
{% /dt %}

{% dd %}
By default, Datadog includes all events between a user's first exposure and the end of the experiment. Use this setting to measure a time-boxed value such as "sessions within 7 days". If you add a time frame filter, the metric only includes events from the specified time window, starting at the moment the experiment first enrolls the user.
{% /dd %}

{% dt %}
Desired metric direction
{% /dt %}

{% dd %}
Datadog highlights statistically significant results. Use this setting to specify whether you want this metric to increase or decrease.
{% /dd %}

{% dt %}
Outlier handling
{% /dt %}

{% dd %}
Real-world data often includes extreme outliers that can impact experiment results. Use this setting to set a threshold at which Datadog truncates data. For example, set a 99% upper bound to truncate all results at the metric's 99th percentile.
{% /dd %}

{% /dl %}

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Make data-driven design decisions with Product Analytics](https://www.datadoghq.com/blog/datadog-product-analytics/)
- [How we built Datadog Experiments](https://www.datadoghq.com/blog/how-we-built-datadog-experiments/)