--- isPrivate: true title: Pipeline Simulation description: >- Use Pipeline Simulation to preview how your processors modify your data before deploying your pipeline configuration. breadcrumbs: Docs > Observability Pipelines > Configuration > Pipeline Simulation --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Pipeline Simulation {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} {% callout %} # Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com {% callout %} ##### Join the Preview! Pipeline Simulation is in Preview. Reach out to your account manager to request access. {% /callout %} {% /callout %} ## Overview{% #overview %} When you configure or edit a pipeline in Observability Pipelines, you often have to update filter queries, sampling rules, or [Packs](https://docs.datadoghq.com/observability_pipelines/packs.md) that transform your telemetry. These changes can impact downstream monitors, dashboards, and detection rules. Therefore, it's important to test and validate how your changes affect your production data before you deploy those changes. Use Pipeline Simulation to preview how your processors, volume control rules, and [Packs](https://docs.datadoghq.com/observability_pipelines/packs.md) modify your data before deploying your pipeline configuration. This helps ensure your processors target the right data and modify your logs as expected. You can test your configuration with live logs sent through the pipeline or import your own sample data. The following example of Pipeline Simulation shows an unparsed log that a processor group receives (Entry) and the parsed output after the log is processed and tagged (Exit). {% image source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_overview.ae827d415b66f3c004480028186aca2b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_overview.ae827d415b66f3c004480028186aca2b.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Pipeline Simulation showing an unparsed log in the Entry column and the parsed output in the Exit column." /%} To use Pipeline Simulation: 1. Capture a snapshot of your data. 1. Run a simulation with that data for your processor configuration. 1. Inspect the data the processor group received and the resulting output. Confirm the processed data is what you expect. If it is not, update your processors and run another simulation. 1. After you validate that the processed data is what you expect, deploy the changes to production. This document goes over each of those steps in detail. ## Permissions{% #permissions %} Only users with the `Observability Pipelines Live Capture Write` permission can set up captures. Users with the `Observability Pipelines Live Capture Read` permission can only view the events that have already been captured. See [Observability Pipelines Permissions](https://docs.datadoghq.com/account_management/rbac/permissions.md#observability-pipelines) for a list of permissions for Observability Pipelines assets. Datadog Admins have read and write permissions by default. Standard users have only read permission by default. See [Access Control](https://docs.datadoghq.com/account_management/rbac.md) for more information about default Datadog roles and how to create custom roles. ### Add domains to firewall allowlist{% #add-domains-to-firewall-allowlist %} If you are using a firewall, add these domains to the allowlist: - `api.` - `obpipeline-intake.` - `config.` ## Capture data for a pipeline simulation{% #capture-data-for-a-pipeline-simulation %} Before running a simulation, you need to capture sample data to test against. Pipeline Simulation lets you capture live data from an active pipeline, reuse previously captured samples, or import your own data. 1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines). 1. [Set up a new pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md?tab=logs) or select an existing pipeline and click Edit Pipeline on the top right side of the page. 1. On the pipeline's edit page, you can expand a processor in a processor group to view its read-only configuration. To edit any processors, capture data, and run a simulation, click Edit on a processor group. {% image source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_edit_processor_group.1ded0b733f34437fd8a3c7b1c5fd547d.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_edit_processor_group.1ded0b733f34437fd8a3c7b1c5fd547d.png?auto=format&fit=max&w=850&dpr=2 2x" alt="A pipeline's edit page with the Edit button on a processor group highlighted." /%} The status of a pipeline determines whether a simulation can be run with live data, imported data, or both. | Pipeline status | Simulation with imported data | Simulation with saved data | Simulation with live data | | --------------- | ----------------------------- | -------------------------- | ------------------------- | | Active | ✔️ | ✔️ | ✔️ | | Inactive | ✔️ | ✔️ | | Draft | ✔️ | ✔️ | **Note**: For inactive pipelines or pipelines in draft mode, you must import data the first time you run a simulation for the pipeline. ### Capture live logs from your pipeline to run a simulation with production data{% #capture-live-logs-from-your-pipeline-to-run-a-simulation-with-production-data %} For active pipelines, Observability Pipelines automatically runs a Live Capture if a previously saved capture, whether from imported data or a live capture, is not available. To run a capture for a specific set of data: 1. Click Live Capture. 1. (Optional) Enter a query to specify which events you want to capture. The default query is `*`. - Use a capture query, such as `service:cloudtrail env:prod`, to target a specific set of logs. This helps to distinguish between different sets of logs you use for your simulations. For example, you can capture a set of logs for custom app log processors and another for the [Cloudflare Pack](https://docs.datadoghq.com/observability_pipelines/packs/cloudflare.md). - For more information on creating queries, see [Search Syntax for Logs](https://docs.datadoghq.com/observability_pipelines/search_syntax/logs.md) or [Search Syntax for Metrics](https://docs.datadoghq.com/observability_pipelines/search_syntax/metrics.md). 1. (Optional) Enter a capture duration (in seconds or minutes) for how long you want events to be captured. - Minimum duration (default, if no duration is specified): 30 seconds - Maximum duration: 300 seconds (5 minutes) 1. Click Capture. ### Import your own sample data for a simulation{% #import-your-own-sample-data-for-a-simulation %} To import sample data for your pipeline simulation: 1. For an active or draft pipeline, click the down arrow next to Live Capture and select Import Sample Data. 1. Drag and drop or browse for your sample data files or paste your sample logs in the modal. **Note**: Sample data must be in a JSON or JSONL file. See Sample file import specifications for more details. 1. Click Confirm. #### Sample file import specifications{% #sample-file-import-specifications %} ##### JSON sample data format{% #json-sample-data-format %} When you import sample data in JSON format, the data must be: - Structured as a set of JSON objects wrapped in an array - Comma-separated between each log event The following is an example set of data in JSON you can import for a simulation: {% tab title="Logs" %} ```json [ { "id": "abcd", "content": { "timestamp": "2026-05-27T19:41:46.609Z", "tags": [ "env:prod", "service:web-store", "source:ruby" ], "host": "123", "service": "web-store", "message": "UnhandledInternalServiceError: Payment service reported 500 error code.", "attributes": { "transaction_id": "abc123", "shopist": { "webstore": { "merchant": { "plan_type": "m2m", "tier": "enterprise", "store_name": "ElectroNova" } }, "customer": { "name": "Pam Jones", "location": "San Francisco, California" } }, "level": "error", "logger": { "name": "ShoppingCartController" }, "status": "error" } } } ] ``` {% /tab %} {% tab title="Metrics" %} ```json [ { "name": "counter", "timestamp": "2026-05-22T12:40:53.761998760Z", "kind": "incremental", "counter": { "value": 1 } }, { "name": "test_metric", "timestamp": "2026-05-22T12:40:53.762001805Z", "kind": "incremental", "distribution": { "samples": [ { "value": 1, "rate": 1 } ], "statistic": "summary" } }, { "name": "counter", "timestamp": "2026-05-22T12:40:53.762002953Z", "kind": "incremental", "counter": { "value": 1 } }, { "name": "test_metric", "timestamp": "2026-05-22T12:40:53.762003454Z", "kind": "incremental", "distribution": { "samples": [ { "value": 1, "rate": 1 } ], "statistic": "summary" } }, { "name": "counter", "timestamp": "2026-05-22T12:40:53.762004406Z", "kind": "incremental", "counter": { "value": 1 } }, { "name": "test_metric", "timestamp": "2026-05-22T12:40:53.762005210Z", "kind": "incremental", "distribution": { "samples": [ { "value": 1, "rate": 1 } ], "statistic": "summary" } }, { "name": "counter", "timestamp": "2026-05-22T12:40:53.762009493Z", "kind": "incremental", "counter": { "value": 1 } }, { "name": "test_metric", "timestamp": "2026-05-22T12:40:53.762010190Z", "kind": "incremental", "distribution": { "samples": [ { "value": 1, "rate": 1 } ], "statistic": "summary" } }, { "name": "counter", "timestamp": "2026-05-22T12:40:53.762010978Z", "kind": "incremental", "counter": { "value": 1 } }, { "name": "test_metric", "timestamp": "2026-05-22T12:40:53.762011618Z", "kind": "incremental", "distribution": { "samples": [ { "value": 1, "rate": 1 } ], "statistic": "summary" } }, { "name": "counter", "timestamp": "2026-05-22T12:40:53.762013275Z", "kind": "incremental", "counter": { "value": 1 } }, { "name": "test_metric", "timestamp": "2026-05-22T12:40:53.762013825Z", "kind": "incremental", "distribution": { "samples": [ { "value": 1, "rate": 1 } ], "statistic": "summary" } }, { "name": "counter", "timestamp": "2026-05-22T12:40:53.762014416Z", "kind": "incremental", "counter": { "value": 1 } }, { "name": "test_metric", "timestamp": "2026-05-22T12:40:53.762015007Z", "kind": "incremental", "distribution": { "samples": [ { "value": 1, "rate": 1 } ], "statistic": "summary" } } ] ``` {% /tab %} ##### JSONL sample data format{% #jsonl-sample-data-format %} A JSONL file must contain multiple JSON objects, where each object is on its own line, separated by a newline. The following is an example JSONL file you can import for a simulation: ```json {"name":"process_payload_ms.count","tags":{"host":"i-abcd123","version":"7.77.3"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}} {"name":"max_eps.max_rate","tags":{"host":"i-efg123","version":"7.77.3"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"absolute","gauge":{"value":200.0}} {"name":"service_checks","tags":{"client_version":"5.8.3","host":"i-hijk123"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}} {"name":"aggregated_context_by_type","tags":{"client_version":"5.8.1","xyz123":"macbook","metrics_type":"distribution"},"timestamp":"2026-05-08T20:16:10Z","interval_ms":10000,"kind":"incremental","counter":{"value":0.0}} ``` ### Reuse previously saved data for your simulation{% #reuse-previously-saved-data-for-your-simulation %} After you use Live Capture to pull data from your pipeline, the results are stored for 72 hours in the Datadog platform and can be used for simulations. To use previously captured data for your pipeline simulation: 1. Click the down arrow next to Live Capture and select See Saved Samples. 1. Click Use as Sample Data for the live capture or previously imported data you want to use. {% image source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_saved_samples.78d2786ec738a3ec992fb21e1ebad0e8.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_saved_samples.78d2786ec738a3ec992fb21e1ebad0e8.png?auto=format&fit=max&w=850&dpr=2 2x" alt="The Saved Samples view with the Use as Sample Data button for a previous capture." /%} ## Run a pipeline simulation{% #run-a-pipeline-simulation %} After you have captured data for your pipeline simulation, edit your pipeline processors and run a simulation. 1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines). 1. Select your pipeline and click Edit Pipeline on the top right side of the page. 1. On the pipeline edit page, click Edit on a processor group to add or edit processors in your pipeline. 1. You can: - Update existing processors in the current group or another group, select the group in the Processor Group dropdown menu. - Click Add Processor to add a processor. - Click + Group to add a processor group. - Click + Pack to add a [Pack](https://docs.datadoghq.com/observability_pipelines/packs.md). - **Note**: For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors. 1. Choose a simulation scenario: - **Run a simulation up to a specific processor group** 1. Keep the Simulate entire pipeline toggle disabled. 1. Click Preview at the bottom right of the processors section to run the simulation. 1. In the Simulated Changes section, select a result to view the data before and after being transformed by all processors in the current processor group and prior groups. See Run a simulation up to a specific processor group for more information. - **Run a simulation up to a specific processor** 1. Keep the Simulate entire pipeline toggle disabled. 1. Click the down arrow on that processor to expand it. 1. Click **Preview Up to This Processor**.In the Simulated Changes section, select a result to view the data before and after being transformed by processors up to the selected processor and the processors prior to this one, including ones in previous processor groups. - **Run a simulation for the entire pipeline** 1. Enable the Simulate entire pipeline toggle. 1. Click Preview at the bottom right of the processors section to run the simulation.In the Simulated Changes section, select a result to view the data before after being transformed by all processors, which is the data sent to the destinations. See Run a simulation for the entire pipeline for more information. {% image source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_preview.7a48eba3b739a80d0007820576c42fdf.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_preview.7a48eba3b739a80d0007820576c42fdf.png?auto=format&fit=max&w=850&dpr=2 2x" alt="The Preview button at the bottom of the processors panel." /%} 1. After the simulation is complete, you can: - See the data that the processor received and sent out. - Enter a free-text search query to find specific logs that were captured. The query searches all attributes and tags in the logs. - Use the All events dropdown menu to view events by status: - `Modified only` shows modified and created events. Created events include: - Metrics generated from logs with the [Generate Logs-Based Metrics](https://docs.datadoghq.com/observability_pipelines/processors/generate_metrics.md) processor. - Logs created by the [Split Array](https://docs.datadoghq.com/observability_pipelines/processors/split_array.md) processor. - `Unmodified only` shows only unmodified events. - `Rerouted only` shows only logs rerouted to an overflow destination by the [Quota processor](https://docs.datadoghq.com/observability_pipelines/processors/quota.md). - `Dropped only` shows only dropped events. - Use the All telemetry dropdown menu to view `Logs only` or `Metrics only` events. `Metrics only` shows events only from the [Generate Logs-Based Metrics](https://docs.datadoghq.com/observability_pipelines/processors/generate_metrics.md) processor. 1. Click Save at the bottom right side of the page to save your changes and then Back to Pipeline on the top right side of the page. 1. To deploy your updates to production, do one of the following: - **Deploy updates using the UI**: Click Next: Deploy Changes on the pipeline page. - **Use the API to deploy changes**: See [Export and Import Configuration](https://docs.datadoghq.com/observability_pipelines/configuration/export_and_import_pipeline_configurations.md) for information on exporting pipeline configurations to JSON. - **Use Terraform to deploy changes**: See [Export and Import Configuration](https://docs.datadoghq.com/observability_pipelines/configuration/export_and_import_pipeline_configurations.md) for information on exporting pipeline configurations to Terraform. **Note**: For the Enrichment Table processor, only the Reference Table option supports Pipeline Simulation. ### Simulation scenarios{% #simulation-scenarios %} When you run a simulation, you can choose to send your data to a specific processor group or through all processor groups to get an end-to-end preview. #### Run a simulation up to a specific processor group{% #run-a-simulation-up-to-a-specific-processor-group %} On the Pipeline Simulation page, you can choose which set of processors you want to test and validate. The simulation sends data through the previous processor groups and the selected group. For example, if you run a simulation for processor group 2, the processed data you see is the result from sending the logs through groups 1 and 2. Similarly, if you run a simulation for group 3, the result is after processing the data through processor groups 1, 2, and 3. {% image source="https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_specific_processor_group.377c8eaecd5cda8b3bcbd3f659a68461.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/pipeline_simulation/pipeline_simulation_specific_processor_group.377c8eaecd5cda8b3bcbd3f659a68461.png?auto=format&fit=max&w=850&dpr=2 2x" alt="The list of processor groups with groups 1, 2, and 3 highlighted." /%} #### Run a simulation for the entire pipeline{% #run-a-simulation-for-the-entire-pipeline %} When you enable the Simulate entire pipeline toggle and run a simulation, the entry event shown is the log sent from the source before any processing is done. The exit log is the result after all processors have been applied and sent to the destination. This lets you visualize how a log is transformed after it goes through the entire pipeline. ## Further reading{% #further-reading %} - [Set up pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md) - [Live Capture](https://docs.datadoghq.com/observability_pipelines/configuration/live_capture.md)