---
isPrivate: true
title: Pipeline Simulation
description: >-
  Use Pipeline Simulation to preview how your processors modify your log data
  before deploying your pipeline configuration.
breadcrumbs: Docs > Observability Pipelines > Configuration > Pipeline Simulation
---

# Pipeline Simulation

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

{% callout %}
# Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com

{% callout %}
##### Join the Preview!

Pipeline Simulation is in Preview. Reach out to your account manager to request access.
{% /callout %}

{% /callout %}

## Overview{% #overview %}

When you configure or edit a pipeline in Observability Pipelines, you often have to update filter queries, sampling rules, or [Packs](https://docs.datadoghq.com/observability_pipelines/packs.md) that transform your telemetry. These changes can impact downstream monitors, dashboards, and detection rules. Therefore, it's important to test and validate how your changes affect your production data before you deploy those changes.

Use Pipeline Simulation to preview how your processors, volume control rules, and [Packs](https://docs.datadoghq.com/observability_pipelines/packs.md) modify your log data before deploying your pipeline configuration. This helps ensure your processors target the right data and modify your logs as expected. You can test your configuration with live logs sent through the pipeline or import your own sample data.

To use Pipeline Simulation:

1. Capture a snapshot of your data.
1. Run a simulation with that data for your processor configuration.
1. Inspect the data the processor group received and the resulting output. Confirm the processed data is what you expect. If it is not, update your processors and run another simulation.
1. After you validate that the processed data is what you expect, deploy the changes to production.

This document goes over each of those steps in detail.

The following example of Pipeline Simulation shows an unparsed log that a processor group receives (Entry) and the parsed output after the log is processed and tagged (Exit).

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_overview.ae827d415b66f3c004480028186aca2b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_overview.ae827d415b66f3c004480028186aca2b.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Pipeline Simulation showing an unparsed log in the Entry column and the parsed output in the Exit column." /%}

## Permissions{% #permissions %}

Only users with the `Observability Pipelines Live Capture Write` permission can set up captures. Users with the `Observability Pipelines Live Capture Read` permission can only view the events that have already been captured. See [Observability Pipelines Permissions](https://docs.datadoghq.com/account_management/rbac/permissions.md#observability-pipelines) for a list of permissions for Observability Pipelines assets.

Datadog Admins have read and write permissions by default. Standard users have only read permission by default. See [Access Control](https://docs.datadoghq.com/account_management/rbac.md) for more information about default Datadog roles and how to create custom roles.

### Add domains to firewall allowlist{% #add-domains-to-firewall-allowlist %}

If you are using a firewall, add these domains to the allowlist:

- `api.`
- `obpipeline-intake.`
- `config.`

## Capture data for a pipeline simulation{% #capture-data-for-a-pipeline-simulation %}

Before running a simulation, you need to capture sample log data to test against. Pipeline Simulation lets you capture live data from an active pipeline, reuse previously captured samples, or import your own data.

1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines).
1. [Set up a pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md?tab=logs) or select a pipeline and click **Edit Pipeline** on the top right side of the page.
1. On the pipeline's edit page, you can expand a processor in a processor group to view its read-only configuration. To edit any processors, capture data, and run a simulation, click **Edit** on a processor group.
   {% image
      source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_edit_processor_group.1ded0b733f34437fd8a3c7b1c5fd547d.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_edit_processor_group.1ded0b733f34437fd8a3c7b1c5fd547d.png?auto=format&fit=max&w=850&dpr=2 2x"
      alt="A pipeline's edit page with the Edit button on a processor group highlighted." /%}

The status of a pipeline determines whether a simulation can be run with live data, imported data, or both.

| Pipeline status | Simulation with imported data | Simulation with saved data | Simulation with live data |
| --------------- | ----------------------------- | -------------------------- | ------------------------- |
| Active          | ✔️                            | ✔️                         | ✔️                        |
| Inactive        | ✔️                            | ✔️                         |
| Draft           | ✔️                            | ✔️                         |

**Note**: For inactive pipelines or pipelines in draft mode, you must import data the first time you run a simulation for the pipeline.

### Capture live logs from your pipeline to run a simulation with production data{% #capture-live-logs-from-your-pipeline-to-run-a-simulation-with-production-data %}

For active pipelines, Observability Pipelines automatically runs a Live Capture if a previously saved capture, whether from imported data or a live capture, is not available.

To run a capture for a specific set of data:

1. Click **Live Capture**.
1. (Optional) Enter a query to specify which events you want to capture. The default query is `*`.
   - Use a capture query, such as `service:cloudtrail env:prod`, to target a specific set of logs. This helps to distinguish between different sets of logs you use for your simulations. For example, you can capture a set of logs for custom app log processors and another for the [Cloudflare Pack](https://docs.datadoghq.com/observability_pipelines/packs/cloudflare.md).
   - For more information on creating queries, see [Search Syntax for Logs](https://docs.datadoghq.com/observability_pipelines/search_syntax/logs.md) or [Search Syntax for Metrics](https://docs.datadoghq.com/observability_pipelines/search_syntax/metrics.md).
1. (Optional) Enter a capture duration (in seconds or minutes) for how long you want events to be captured.
   - Minimum duration (default, if no duration is specified): 30 seconds
   - Maximum duration: 300 seconds (5 minutes)
1. Click **Capture**.

### Import your own sample data for a simulation{% #import-your-own-sample-data-for-a-simulation %}

To import sample data for your pipeline simulation:

1. For an inactive or draft pipeline, if the pipeline does not have saved data, click **Import Data**. Otherwise, click the down arrow next to **Live Capture** and select **Import Sample Data**.
1. In the modal, drag and drop your sample data files or browse for them, or click the **Paste** tab to paste your sample data. **Note**: Sample data must be in a JSON or JSONL file. See Sample file import specifications for more details.
1. Click **Confirm**.

#### Sample file import specifications{% #sample-file-import-specifications %}

##### JSON sample data format{% #json-sample-data-format %}

When you import sample data in JSON format, the data must be:

- Structured as a set of JSON objects wrapped in an array
- Comma-separated between each log event

The following is an example set of logs in JSON you can import for a simulation:

```json
[
  {
    "id": "abcd",
    "content": {
      "timestamp": "2026-05-27T19:41:46.609Z",
      "tags": [
        "env:prod",
        "service:web-store",
        "source:ruby"
      ],
      "host": "123",
      "service": "web-store",
      "message": "UnhandledInternalServiceError: Payment service reported 500 error code.",
      "attributes": {
        "transaction_id": "abc123",
        "shopist": {
          "webstore": {
            "merchant": {
              "plan_type": "m2m",
              "tier": "enterprise",
              "store_name": "ElectroNova"
            }
          },
          "customer": {
            "name": "Pam Jones",
            "location": "San Francisco, California"
          }
        },
        "level": "error",
        "logger": {
          "name": "ShoppingCartController"
        },
        "status": "error"
      }
    }
  }
]
```

##### JSONL sample data format{% #jsonl-sample-data-format %}

A JSONL file must contain multiple JSON objects, where each object is on its own line, separated by a newline.

The following is an example JSONL file you can import for a simulation:

```json
{"name":"process_payload_ms.count","tags":{"host":"i-abcd123","version":"7.77.3"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}}
{"name":"max_eps.max_rate","tags":{"host":"i-efg123","version":"7.77.3"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"absolute","gauge":{"value":200.0}}
{"name":"service_checks","tags":{"client_version":"5.8.3","host":"i-hijk123"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}}
{"name":"aggregated_context_by_type","tags":{"client_version":"5.8.1","xyz123":"macbook","metrics_type":"distribution"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}}
```

### Reuse previously saved data for your simulation{% #reuse-previously-saved-data-for-your-simulation %}

After you use Live Capture to pull data from your pipeline, the results are stored for 72 hours in the Datadog platform and can be used for simulations. To use previously captured data for your pipeline simulation:

1. Click the down arrow next to **Live Capture** and select **See Saved Samples**.
1. Click **Use as Sample Data** for the live capture or previously imported data you want to use.

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_saved_samples.78d2786ec738a3ec992fb21e1ebad0e8.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_saved_samples.78d2786ec738a3ec992fb21e1ebad0e8.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Saved Samples view with the Use as Sample Data button for a previous capture." /%}

## Run a pipeline simulation{% #run-a-pipeline-simulation %}

After you have captured data for your pipeline simulation, edit your pipeline processors and run a simulation.

1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines).
1. Select your pipeline and click **Edit Pipeline** on the top right side of the page.
1. On the pipeline edit page, click **Edit** on a processor group to add or edit processors in your pipeline.
1. In the **Processor Group** dropdown menu on the left side of the Pipeline Simulation page, select the processor group with the processors you want to update and test. You can hover over **Processor Group** to see a snapshot of where the processors are in the pipeline.
   {% video
      url="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/processor_group_map.mp4" /%}
1. Add and update your processors. You can update multiple processors for one simulation. **Note**: For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors.
1. Choose a simulation scenario:
   - **Run a simulation up to a specific processor group**: Keep the **End-to-End Preview** toggle disabled to see the result of logs transformed by all processors in the current processor group and the prior processor groups. See Run a simulation up to a specific processor group for more information.
   - **Run an end-to-end preview simulation**: Enable the **End-to-End Preview** toggle to run a simulation through all processor groups and see the data that is sent to the destination. See End-to-end preview for more information.
     {% image
        source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_end_to_end_toggle.d7127f05f78c5f75fb6bf4a8e814f2d2.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_end_to_end_toggle.d7127f05f78c5f75fb6bf4a8e814f2d2.png?auto=format&fit=max&w=850&dpr=2 2x"
        alt="The End-to-End Preview toggle on the Pipeline Simulation page." /%}
1. Click **Preview** at the bottom of the processors panel to run the simulation.
   {% image
      source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_preview.7a48eba3b739a80d0007820576c42fdf.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_preview.7a48eba3b739a80d0007820576c42fdf.png?auto=format&fit=max&w=850&dpr=2 2x"
      alt="The Preview button at the bottom of the processors panel." /%}
1. After the simulation is complete, you can:
   - See the log data that the processor received and sent out.
   - Enter a free-text search query to find specific logs that were captured. The query searches all attributes and tags in the logs.
   - Use the **All events** dropdown menu to view events by status:
     - `Modified only` shows only modified events.
     - `Unmodified only` shows only unmodified events.
     - `Rerouted only` shows only logs rerouted to an overflow destination by the [Quota processor](https://docs.datadoghq.com/observability_pipelines/processors/quota.md).
     - `Created only` shows only:
       - Metrics generated from logs with the [Generate Logs-Based Metrics](https://docs.datadoghq.com/observability_pipelines/processors/generate_metrics.md) processor.
       - Logs created by the [Split Array](https://docs.datadoghq.com/observability_pipelines/processors/split_array.md) processor.
     - `Dropped only` shows only dropped events.
   - Use the **All telemetry** dropdown menu to view `Logs only` or `Metrics only` events. `Metrics only` shows events only from the [Generate Logs-Based Metrics](https://docs.datadoghq.com/observability_pipelines/processors/generate_metrics.md) processor.
     {% video
        url="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/view_simulation.mp4" /%}
1. Click **Save** at the bottom right side of the page to save your changes.
1. Click **Back to Pipeline** on the top right side of the page.
1. To deploy your updates to production, do one of the following:
   - **Deploy updates using the UI**: Click **Next: Deploy Changes** on the pipeline page.
   - **Use the API to deploy changes**: See [Export and Import Configuration](https://docs.datadoghq.com/observability_pipelines/configuration/export_and_import_pipeline_configurations.md) for information on exporting pipeline configurations to JSON.
   - **Use Terraform to deploy changes**: See [Export and Import Configuration](https://docs.datadoghq.com/observability_pipelines/configuration/export_and_import_pipeline_configurations.md) for information on exporting pipeline configurations to Terraform.

**Note**: For the Enrichment Table processor, only the Reference Table option supports Pipeline Simulation.

### Simulation scenarios{% #simulation-scenarios %}

When you run a simulation, you can choose to send your data to a specific processor group or through all processor groups to get an end-to-end preview.

#### Run a simulation up to a specific processor group{% #run-a-simulation-up-to-a-specific-processor-group %}

On the Pipeline Simulation page, you can choose which set of processors you want to test and validate. The simulation sends data through the previous processor groups and the selected group. For example, if you run a simulation for processor group 2, the processed data you see is the result from sending the logs through groups 1 and 2. Similarly, if you run a simulation for group 3, the result is after processing the log data through processor groups 1, 2, and 3.

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_specific_processor_group.377c8eaecd5cda8b3bcbd3f659a68461.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_specific_processor_group.377c8eaecd5cda8b3bcbd3f659a68461.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The list of processor groups with groups 1, 2, and 3 highlighted." /%}

#### End-to-end preview{% #end-to-end-preview %}

When you toggle **End-to-End Preview** and run a simulation, the entry event shown is the log sent from the source before any processing is done. The exit log is the result after all processors have been applied and sent to the destination. This lets you visualize how a log is transformed after it goes through the entire pipeline.

## Further reading{% #further-reading %}

- [Set up pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md)
- [Live Capture](https://docs.datadoghq.com/observability_pipelines/configuration/live_capture.md)