Overview

Warehouse-native experiment analysis lets you run statistical computations directly in your data warehouse.

To set this up for Amazon Redshift, connect a Redshift cluster to Datadog using the AWS integration and configure your experiment settings. This guide covers:

• Preparing the Redshift cluster
• Creating AWS resources and granting IAM permissions
• Configuring experiment settings in Datadog

Prerequisites

Datadog Experiments connects to Redshift through Datadog’s Amazon Web Services (AWS) integration. If you already have the AWS integration configured for the account containing your Redshift cluster, skip to Step 1.

Step 1: Prepare the Redshift cluster

Create a Datadog service user and a dedicated schema for Datadog to store experiment results and intermediate tables.

Create a Datadog service user in your Redshift database

Run the following command to create a service user with a strong password that Datadog can use to execute queries. Replace datadog_experiments_user with your user value and Your_Strong_Password with your password.

CREATE USER datadog_experiments_user PASSWORD 'Your_Strong_Password';

Create an output schema

Run the following commands to create a schema where Datadog can store experiment results and intermediate tables. Replace datadog_experiments_output with your schema name and datadog_experiments_user with your service user value.

CREATE SCHEMA IF NOT EXISTS datadog_experiments_output;
GRANT ALL ON SCHEMA datadog_experiments_output TO datadog_experiments_user;

Grant the service user read access to your metric data

Grant the service user read access to the tables or schemas that contain your source data. These are the tables you plan to use for experiment metrics, and are typically in a different schema than the output schema created above. Run the GRANT USAGE command, then run the GRANT SELECT option that matches your access needs. Replace datadog_experiments_user, <schema>, and <table> with the appropriate values.

GRANT USAGE ON SCHEMA <schema> TO datadog_experiments_user;

-- Option 1: Give read access to a single table
GRANT SELECT ON TABLE <schema>.<table> TO datadog_experiments_user;

-- Option 2: Give read access to all tables in the schema
GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO datadog_experiments_user;

Step 2: Create AWS resources and grant IAM permissions

Create an S3 bucket

Create an S3 bucket for importing exposure events into your warehouse. The bucket name must start with datadog-experimentation- (for example, datadog-experimentation-[aws_account_id]). You can use the bucket’s default settings.

Grant additional IAM permissions

In addition to the permissions listed in the AWS integration documentation, Datadog Experiments requires additional IAM permissions to run warehouse-native experiment analysis.

Use the following table to gather the values for your environment, then add the policy statement below to the IAM role that your Datadog AWS integration uses.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RedshiftGetClusterCredentials",
      "Effect": "Allow",
      "Action": [
        "redshift:GetClusterCredentials"
      ],
      "Resource": [
        "[Redshift cluster ARN]",
        "[Redshift user ARN]",
        "[Redshift database ARN]"
      ]
    },
    {
      "Sid": "QueryRedshift",
      "Effect": "Allow",
      "Action": [
        "redshift-data:ExecuteStatement",
        "redshift-data:GetStatementResult",
        "redshift-data:DescribeStatement",
        "redshift-data:ListStatements",
        "redshift-data:CancelStatement"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ListExperimentationBucket",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "[S3 bucket ARN]"
    },
    {
      "Sid": "ReadWriteExperimentationBucket",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "[S3 bucket ARN]/*"
    }
  ]
}

Step 3: Configure experiment settings

After you set up your AWS integration and Redshift cluster, configure the experiment settings in Datadog:

1. Open Datadog Product Analytics.
2. In the left navigation, hover over Settings and click Experiments.
3. Select the Warehouse Connections tab.
4. Click Connect a data warehouse. If you already have a warehouse connected, click Edit instead.
5. Select the Redshift tile.
6. Select your AWS account from the dropdown.
7. Under Cluster Connection, enter:
   • AWS region: The region your Redshift cluster is in (for example, us-east-1).
   • Cluster identifier: The name of your Redshift cluster.
   • Cluster endpoint: The full endpoint URL for your cluster.
   • Port: The port your cluster is listening on (default: 5439).
8. Under Database and Storage, enter:
   • Database: The name of the database containing your source tables.
   • Database user: The service user you created in Step 1 (for example, datadog_experiments_user).
   • Schema: The schema you created in Step 1 for Datadog Experiments to write to (for example, datadog_experiments_output).
   • Temp S3 bucket: The S3 bucket you created in Step 2 (for example, datadog-experimentation-[aws_account_id]).
9. Click Save.

After you save your warehouse connection, create experiment metrics using your Redshift data.

Further reading

Additional helpful documentation, links, and articles:

Defining metrics in Datadog ExperimentsDOCUMENTATION How to bridge speed and quality in experiments through unified dataBLOG