--- isPrivate: true title: Export Pipelines description: >- Forward your RUM and Product Analytics events to your own cloud storage for warehouse loading or long-term retention. breadcrumbs: Docs > RUM & Session Replay > Export Pipelines --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Export Pipelines {% callout %} ##### Join the Preview! Export Pipelines is in Preview. [Request Access](https://www.datadoghq.com/product-preview/export-pipelines/) {% /callout %} ## Overview{% #overview %} Export Pipelines stream your ingested Real User Monitoring (RUM) and Product Analytics events to a cloud storage bucket you own — Amazon S3, Azure Blob Storage, or Google Cloud Storage — in JSON or Parquet format. {% image source="https://docs.dd-static.net/images/real_user_monitoring/rum_export/rum-export-overview.2b77b5cc2de3332c3a5f64536796f2ba.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/real_user_monitoring/rum_export/rum-export-overview.2b77b5cc2de3332c3a5f64536796f2ba.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Export Pipelines list on the RUM application settings page" /%} Use Export Pipelines to: - Load your event data into your own data warehouse (such as Snowflake, BigQuery, or Databricks) for custom analytics and reporting. - Meet compliance requirements or retain raw events for long-term archival and ad-hoc investigations. Datadog only manages the export from your Datadog account to your cloud storage system. ## How it works{% #how-it-works %} Export Pipelines is a shared feature between [Real User Monitoring](https://docs.datadoghq.com/real_user_monitoring.md) and [Product Analytics](https://docs.datadoghq.com/product_analytics.md). Pipelines are configured at two different levels: | Scope | Source | Max pipelines | Available presets | | ---------------- | ----------------- | ------------- | ------------------------------------------------------------------------------ | | Per application | RUM | 1 | *All RUM event types*, or *Sessions, Views & Actions only* | | Per application | Product Analytics | 1 | *All Product Analytics Events* (sessions, views, actions & server-side events) | | Per organization | Product Analytics | 1 | *User and Account Profiles* | The *User and Account Profiles* preset is limited to one pipeline per organization. Because user and profile data is shared across all your applications, creating one pipeline per application would result in duplicate records in your storage. Each pipeline exports continuously and independently of the others. ## Prerequisites{% #prerequisites %} - RUM is enabled on the application (or Product Analytics, or both). - The Datadog integration for your cloud provider is set up: [Amazon Web Services](https://docs.datadoghq.com/integrations/amazon-web-services.md), [Azure](https://docs.datadoghq.com/integrations/azure.md), or [Google Cloud](https://docs.datadoghq.com/integrations/google_cloud_platform.md). - Your Datadog user has the `rum_write_archives` permission. See [Role Based Access Control](https://docs.datadoghq.com/account_management/rbac/permissions.md). ## Set up an export pipeline{% #set-up-an-export-pipeline %} ### 1. Set up your cloud integration {% tab title="AWS S3" %} If not already configured, set up the [AWS integration](https://docs.datadoghq.com/integrations/amazon_web_services.md?tab=automaticcloudformation#setup) for the account that holds your S3 bucket. This usually means creating a role that Datadog can assume. For AWS China accounts, use access keys instead of role delegation. {% /tab %} {% tab title="Azure Storage" %} Set up the [Azure integration](https://app.datadoghq.com/account/settings#integrations/azure) in the subscription that holds your storage account, if you haven't already. This involves [creating an app registration that Datadog can use](https://docs.datadoghq.com/integrations/azure.md?tab=azurecliv20#integrating-through-the-azure-portal). **Note:** Exporting to Azure ChinaCloud and Azure GermanyCloud is not supported. Exporting to Azure GovCloud is supported in Preview — contact Datadog support to request access. {% /tab %} {% tab title="Google Cloud Storage" %} Set up the [Google Cloud integration](https://app.datadoghq.com/account/settings#integrations/google-cloud-platform) for the project that holds your GCS bucket, if you haven't already. This involves [creating a Google Cloud service account that Datadog can use](https://docs.datadoghq.com/integrations/google_cloud_platform.md?tab=datadogussite#setup). {% /tab %} ### 2. Create a storage bucket {% tab title="AWS S3" %} In the [AWS console](https://s3.console.aws.amazon.com/s3), [create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) for your exports. **Notes:** - Do not make the bucket publicly readable. - For [US1, US3, and US5 sites](https://docs.datadoghq.com/getting_started/site.md), see [AWS pricing](https://aws.amazon.com/s3/pricing/) for inter-region data transfer fees. Consider creating the bucket in `us-east-1` to minimize transfer costs. {% /tab %} {% tab title="Azure Storage" %} - In the [Azure Portal](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts), [create a storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal). Choose **Standard** performance or **Block blobs premium**, and select either the **hot** or **cool** access tier. - Create a **container** inside that storage account. Note the container name — you reference it when configuring the pipeline. **Note:** Do not set [immutability policies](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-immutability-policies-manage). Some events occasionally need to be rewritten (typically on retry after a timeout). {% /tab %} {% tab title="Google Cloud Storage" %} In the [Google Cloud console](https://console.cloud.google.com/storage), [create a GCS bucket](https://cloud.google.com/storage/docs/quickstart-console) for your exports. Under **Choose how to control access to objects**, select **Set object-level and bucket-level permissions.** **Note:** Do not add a [retention policy](https://cloud.google.com/storage/docs/bucket-lock). Some events occasionally need to be rewritten (typically on retry after a timeout). {% /tab %} ### 3. Grant Datadog access to the bucket {% tab title="AWS S3" %} 1. [Create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html) with the following statements: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DatadogExportPipelineFiles", "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": [ "arn:aws:s3:::/*", "arn:aws:s3:::/*" ] } ] } ``` - `PutObject` is required to upload export files. - `GetObject` is required to run **Test Configuration**, which writes a test file and reads it back to verify access. - The resource value must end with `/*` — these permissions apply to objects inside the buckets. 1. Edit the bucket names. 1. Optionally, restrict the policy to specific paths. 1. Attach the policy to the Datadog integration role: - In the AWS IAM console, go to **Roles** and open the role used by the Datadog integration. By default it is named `DatadogIntegrationRole`, but the name may differ if your organization renamed it. - Click **Add permissions**, then **Attach policies**. - Enter the name of the policy you just created. - Click **Attach policies**. {% /tab %} {% tab title="Azure Storage" %} 1. Grant the Datadog app permission to write to your storage account. 1. On the [Storage accounts page](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts), select your storage account, open **Access Control (IAM)**, and choose **Add → Add Role Assignment**. 1. Assign the **Storage Blob Data Contributor** role to the Datadog app you created when integrating with Azure, then save. {% /tab %} {% tab title="Google Cloud Storage" %} 1. Grant your Datadog Google Cloud service account permission to write to your bucket. 1. On the [Google Cloud IAM Admin page](https://console.cloud.google.com/iam-admin/iam), select your Datadog service account principal and click **Edit principal**. 1. Click **ADD ANOTHER ROLE**, select **Storage Object Admin**, and save. {% /tab %} **Note:** If your bucket restricts network access by IP, add the webhook IP ranges from the IP ranges list to the allowlist. ### 4. Configure the pipeline in Datadog 1. In Datadog, navigate to **Digital Experience > Real User Monitoring > Manage Applications**. 1. Select your application, then go to **Routing > Export Pipelines**. 1. Click **New Export Pipeline**. 1. Fill in each section of the side panel: **Define Data to Export** Choose the data you want to export. The source and preset options are clearly labeled in the UI. **File format** {% image source="https://docs.dd-static.net/images/real_user_monitoring/rum_export/rum-export-data-format.8bdd5ce1ae57140e010c5bcffe7e682b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/real_user_monitoring/rum_export/rum-export-data-format.8bdd5ce1ae57140e010c5bcffe7e682b.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Data selection and file format options in the Export Pipeline configuration panel" /%} - **Parquet**: best for loading directly into a data warehouse such as BigQuery, Snowflake, or Databricks. - **JSON**: best when you want to process the data with your own pipeline. **Select Storage Type and Configure Bucket** | Provider | Fields | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | **Amazon S3** | AWS account and role (from your AWS integration); **Bucket** (required); **Path** (optional prefix) | | **Azure Blob Storage** | Azure tenant and client (from your Azure integration); **Storage Account** (required); **Container** (required); **Path** (optional prefix) | | **Google Cloud Storage** | GCP service account (from your Google Cloud integration); **Bucket** (required); **Path** (optional prefix) | 1. Optionally, click **Test Configuration**. Datadog writes a small test file to your bucket and reads it back to verify access. Fix any reported permissions or naming issues before saving. 1. Click **Add Export Pipeline** to start the pipeline. ## Pipeline statuses{% #pipeline-statuses %} Once created, a pipeline shows one of these states on the Export Pipelines page: | Status | Meaning | | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Active | Datadog is exporting events successfully. | | Pending | The pipeline was just created or updated. Allow a few minutes before the first upload. | | Error | Datadog could not write to the bucket — typically a permissions or naming issue. Open the pipeline and run **Test Configuration** to see details. | | Paused | The pipeline is disabled. No events are exported. | If a pipeline stays in **Pending** for more than 15 minutes, run **Test Configuration** to surface the underlying issue. ## File format{% #file-format %} Files are organized in a directory structure that makes it easy to query archives by date: ``` //dt=/hour=/archive_.. ``` | Format | Extension | Notes | | ------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | JSON | `.json.gz` | Gzip-compressed newline-delimited JSON. | | Parquet | `.parquet` | Native Parquet encoding (no gzip wrapper). One row per event with typed columns. Directly loadable by Snowflake, BigQuery, and Databricks without preprocessing. | ## Further reading{% #further-reading %} - [Learn about the RUM Explorer](https://docs.datadoghq.com/real_user_monitoring/explorer.md) - [Learn about RUM without Limits](https://docs.datadoghq.com/real_user_monitoring/rum_without_limits.md) - [Learn about Product Analytics](https://docs.datadoghq.com/product_analytics.md)