Overview

Define the metrics you want to measure during your experimentation. Metrics can be built using Product Analytics and Real User Monitoring (RUM) data.

Create your first metric

To create a metric for your experiment:

1. Navigate to the Metrics page within Datadog Product Analytics.
2. Click + Create Metric at the top right corner.
3. Click Select an Event to see a list of all of the actions and views collected from the Datadog SDK.
4. Add a metric name and optionally a description. Then, click Save.

Specify metric aggregation

Default metric aggregation

After you’ve selected your event of interest, you can specify an aggregation method. Metrics default to a count of unique subjects (often users) that had at least one event. However, you can also choose to count the total number of events, or sum a property of that event:

Default metric normalization

All metrics are normalized by the number of enrolled subjects. For example, a count of unique users metric is computed as:

                      Number of users with specified event
                  -----------------------------------------------           
                      Number of users enrolled into this variant

Similarly, a Sum of metric is computed as:

                  Sum of property over users enrolled in this variant
              --------------------------------------------------------           
                      Number of users enrolled into this variant

Custom metric normalization

You can also choose to normalize metrics by a different denominator. To do this, click on a metric then on Create ratio. This allows you to normalize your metric by another event, counting either the number of subjects with that event, the total number of events, or the sum of an event property.

For example, an e-commerce company that wants to measure the Average Order Value can create a ratio metric with the sum of purchase revenue as the numerator, and the count of purchase events as the denominator.

Datadog’s statistical engine accounts for correlations between the numerator and denominator using the delta method.

Add filters

You can also add filters to your metrics, similar to other Product Analytics dashboards. For instance, you might want to filter page views based on referring URL or UTM parameters. Similarly, you might want to filter actions to a specific page or value of a custom attribute. As you add filters, you can check metric values in real time using the chart on the right.

Advanced options

Datadog supports several advanced options specific to experimentation:

Timeframe filters

• By default, Datadog will include all events between a user’s first exposure and the end of the experiment. If you want to measure a time-boxed value such as “sessions within 7 days”, you can add a timeframe filter.
• If selected, the metric will only include events from the specified time window, starting at the moment the user was first enrolled.

Desired metric change

• Datadog highlights statistically significant results.
• Use this setting to specify whether an increase or decrease in this metric is desired.

Guardrail cutoffs

• Use this to specify a do no harm threshold.
• Metrics will be yellow until the confidence interval is small enough to rule out a downside risk larger than this threshold. This indicates that you should continue running your experiment to rule out an effect worse than the number entered here.

Outlier handling

• Real world data often includes extreme outliers that can impact experiment results.
• Use this setting to set a threshold at which data is truncated. For instance, set a 99% upper bound to truncate all results at the metric’s 99th percentile.

Further reading

Documentation, liens et articles supplémentaires utiles:

Make data-driven design decisions with Product AnalyticsBLOG Reading Experiment ResultsDOCUMENTATION