---
title: Set Up Pipelines
description: >-
  Learn how to set up an Observability Pipelines pipeline's source, processors,
  and destinations.
breadcrumbs: Docs > Observability Pipelines > Configuration > Set Up Pipelines
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Set Up Pipelines

## Overview{% #overview %}

{% alert level="info" %}
The pipelines and processors outlined in this documentation are specific to on-premises logging environments. To aggregate, process, and route cloud-based logs, see [Log Management Pipelines](https://docs.datadoghq.com/logs/log_configuration/pipelines.md?tab=source).
{% /alert %}

In Observability Pipelines, a pipeline is a sequential path with three types of components:

- [Sources](https://docs.datadoghq.com/observability_pipelines/sources.md): Receive data from your data sources (for example, the Datadog Agent).
- [Processors](https://docs.datadoghq.com/observability_pipelines/processors.md): Enrich and transform your data.
- [Destinations](https://docs.datadoghq.com/observability_pipelines/destinations.md): Where your processed data is sent.

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/archive_log_pipeline.4ec6cfc248e84c55b569425ac3328fd5.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/archive_log_pipeline.4ec6cfc248e84c55b569425ac3328fd5.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Pipeline with one source connected to two processor groups and two destinations" /%}

You can create a pipeline with one of the following methods:

- Pipeline UI
- API
- Terraform

See [Export a Pipeline Configuration to JSON or Terraform](https://docs.datadoghq.com/observability_pipelines/configuration/export_pipeline_configuration.md) if you want to programmatically deploy a pipeline created in the UI.

## Set up a pipeline in the UI{% #set-up-a-pipeline-in-the-ui %}

### Set up pipeline components{% #set-up-pipeline-components %}

{% tab title="Logs" %}

1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines).
1. Select a [template](https://docs.datadoghq.com/observability_pipelines/configuration/explore_templates.md) based on your use case, or click New Pipeline and select Logs Pipeline.
1. Select and set up a [log source](https://docs.datadoghq.com/observability_pipelines/sources.md?tab=logs#sources). To add more sources, see Add another source.
1. Select and set up [destinations](https://docs.datadoghq.com/observability_pipelines/destinations.md?tab=logs#destinations) for your processed logs.
1. Click Edit on a processor group to add, update, and validate your [processors](https://docs.datadoghq.com/observability_pipelines/processors.md) using [Pipeline Simulation](https://docs.datadoghq.com/observability_pipelines/configuration/pipeline_simulation.md).
   - **Notes**:
     - For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors.
     - If you want to copy a processor, click the copy icon for that processor and then use `command-v` to paste it.

#### Add another source{% #add-another-log-source %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_source.0214877a5a733f7ff5a52dc9bf4006e6.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_source.0214877a5a733f7ff5a52dc9bf4006e6.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing two sources sending logs to one processor group and one destination." /%}

To add another source to your pipeline:

1. Click **Add another source** at the bottom of the Sources box.
1. Select a source from the dropdown menu and set it up.

**Notes**:

- You can add a total of 10 sources for a pipeline.
- If you add multiple sources of the same type to a pipeline, you must use [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md). For example, if you add two HTTP Client sources for two different HTTP clients, you must use secret identifiers for the HTTP client endpoint URL. You cannot use the default `SOURCE_HTTP_CLIENT_ENDPOINT_URL` to store two different HTTP client URIs.

#### Add another processor group{% #add-another-processor-group %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_processor_group.ecdee1d75efe73107cd0cd628052da1f.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_processor_group.ecdee1d75efe73107cd0cd628052da1f.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing two processor groups sending logs to the same destination" /%}

To add another group of processors for a destination:

1. Click the plus sign (**+**) at the bottom of the existing processor group.
1. Click the name of the processor group to update it.
1. Optionally, enter a group filter. Only data that match the filter query are sent through the processors in the group. See [Search Syntax](https://docs.datadoghq.com/observability_pipelines/search_syntax/logs.md) for more information.
1. Click **Add** to add processors to the group.
1. If you want to copy all processors in a group and paste them into the same processor group or a different group:
   1. Click the three dots on the processor group.
   1. Select **Copy all processors**.
   1. Select the desired processor group, and then paste the processors into it.
1. You can toggle the switch to enable and disable the processor group and also each individual processor.

**Notes**:

- There is a limit of 25 processor groups for a pipeline canvas.

#### Add another set of processors and destinations{% #add-another-set-of-processors-and-destinations %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_set_processor_destination.c9a4b6845d6091f1ccf16cd04eaaee35.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_set_processor_destination.c9a4b6845d6091f1ccf16cd04eaaee35.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing two processor groups sending logs to two different destinations" /%}

To add another set of processors and destinations, click the plus sign (**+**) to the left of the processor group to add another set of processors and destinations to the source.

To delete a processor group, you need to delete all destinations linked to that processor group. When the last destination is deleted, the processor group is removed with it.

#### Add another destination to a processor group{% #add-another-destination-to-a-processor-group %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_destination.93229910fa7394f93c6c6ff68aa42fd2.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_destination.93229910fa7394f93c6c6ff68aa42fd2.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing one processor group sending logs to two different destinations" /%}

To add an additional destination to a processor group, click the plus sign (**+**) to the right of the processor group.

To delete a destination, click on the trash icon to the top right of the destination.

- If you delete a destination from a processor group that has multiple destinations, only the deleted destination is removed.
- If you delete a destination from a processor group that only has one destination, both the destination and the processor group are removed.

You must delete the destination to stop the Worker from sending events to it. Disabling all processors associated with the destination does not disable it. The Worker continues to send events to it.

**Notes**:

- A pipeline must have at least one destination. If a processor group only has one destination, that destination cannot be deleted.
- You can add a total of 20 destinations for a pipeline.
- If you add multiple destinations of the same type to a pipeline, you must use [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md). For example, if you add two HTTP Client destinations for two different HTTP clients, you must use secret identifiers for the HTTP client URIs. You cannot use the default `DESTINATION_HTTP_CLIENT_URI` to store the two different HTTP client URIs.

{% /tab %}

{% tab title="Metrics" %}

1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines).
1. Select the [Metric Tag Governance](https://docs.datadoghq.com/observability_pipelines/configuration/explore_templates.md?tab=metrics#metric-tag-governance) template, or click New Pipeline and select Metrics Pipeline.
1. Select and set up a [metrics source](https://docs.datadoghq.com/observability_pipelines/sources.md?tab=metrics#sources). To add more sources, see Add another source.
1. Select and set up [destinations](https://docs.datadoghq.com/observability_pipelines/destinations.md?tab=metrics#destinations) for your processed metrics.
1. Click Edit on a processor group to add, update, and validate your [processors](https://docs.datadoghq.com/observability_pipelines/processors.md?tab=metrics#processors) using [Pipeline Simulation](https://docs.datadoghq.com/observability_pipelines/configuration/pipeline_simulation.md).
   - **Notes**:
     - For a pipeline canvas, there is a limit of 25 processor groups and a total of 150 processors.
     - If you want to copy a processor, click the copy icon for that processor and then paste it (`Cmd+V` on Mac, `Ctrl+V` on Windows/Linux).

#### Add another source{% #add-another-metrics-source %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_source_metrics.2a7a1f3c588e75eeaef3a733b9631d53.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_source_metrics.2a7a1f3c588e75eeaef3a733b9631d53.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing two sources sending metrics to one processor group and one destination." /%}

To add another source to your pipeline:

1. Click **Add another source** at the bottom of the Sources box.
1. Select a source from the dropdown menu and set it up.

**Notes**:

- You can add a total of 10 sources for a pipeline.
- If you add multiple sources of the same type to a pipeline, you must use [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md). For example, if you add two HTTP Client sources for two different HTTP clients, you must use secret identifiers for the HTTP client endpoint URL. You cannot use the default `SOURCE_HTTP_CLIENT_ENDPOINT_URL` to store two different HTTP client URIs.

#### Add another processor group{% #add-another-processor-group %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_processor_group_metrics.c25cab1c21073c5c57e4889516100dc4.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_processor_group_metrics.c25cab1c21073c5c57e4889516100dc4.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing two processor groups sending metrics to the same destination" /%}

To add another group of processors for a destination:

1. Click the plus sign (**+**) at the bottom of the existing processor group.
1. Click the name of the processor group to update it.
1. Optionally, enter a group filter. Only data that match the filter query are sent through the processors in the group. See [Search Syntax](https://docs.datadoghq.com/observability_pipelines/search_syntax/metrics.md) for more information.
1. Click **Add** to add processors to the group.
1. If you want to copy all processors in a group and paste them into the same processor group or a different group:
   1. Click the three dots on the processor group.
   1. Select **Copy all processors**.
   1. Select the desired processor group, and then paste the processors into it.
1. You can toggle the switch to enable and disable the processor group and also each individual processor.

**Notes**:

- There is a limit of 25 processor groups for a pipeline canvas.

#### Add another set of processors and destinations{% #add-another-set-of-processors-and-destinations %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_set_processor_destination_metrics.66f6335ae159f506c11d1b37b4875212.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_set_processor_destination_metrics.66f6335ae159f506c11d1b37b4875212.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing two processor groups sending metrics to two different destinations" /%}

To add another set of processors and destinations, click the plus sign (**+**) to the left of the processor group to add another set of processors and destinations to the source.

To delete a processor group, you need to delete all destinations linked to that processor group. When the last destination is deleted, the processor group is removed with it.

#### Add another destination to a processor group{% #add-another-destination-to-a-processor-group %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/another_destination_metrics.811ac3afb8326f2678e2ab73840397e9.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/another_destination_metrics.811ac3afb8326f2678e2ab73840397e9.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Pipelines page showing one processor group sending metrics to two different destinations" /%}

To add an additional destination to a processor group, click the plus sign (**+**) to the right of the processor group.

To delete a destination, click on the trash icon to the top right of the destination.

- If you delete a destination from a processor group that has multiple destinations, only the deleted destination is removed.
- If you delete a destination from a processor group that only has one destination, both the destination and the processor group are removed.

You must delete the destination to stop the Worker from sending events to it. Disabling all processors associated with the destination does not disable it. The Worker continues to send events to it.

**Notes**:

- A pipeline must have at least one destination. If a processor group only has one destination, that destination cannot be deleted.
- You can add a total of 20 destinations for a pipeline.
- If you add multiple destinations of the same type to a pipeline, you must use [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md). For example, if you add two HTTP Client destinations for two different HTTP clients, you must use secret identifiers for the HTTP client URIs. You cannot use the default `DESTINATION_HTTP_CLIENT_URI` to store the two different HTTP client URIs.

{% /tab %}

### Install the Worker and deploy the pipeline{% #install-the-worker-and-deploy-the-pipeline %}

After you have set up your source, processors, and destinations, click Next: Install. See [Install the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md) for instructions on how to install the Worker for your platform. See [Advanced Worker Configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md) for bootstrapping options.

If you want to make changes to your pipeline after you have deployed it, see [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md?).

### Enable out-of-the-box monitors for your pipeline{% #enable-out-of-the-box-monitors-for-your-pipeline %}

1. Navigate to the [Pipelines](https://app.datadoghq.com/observability-pipelines) page and find your pipeline.
1. Click Enable monitors in the Monitors column for your pipeline.
1. Click Start to set up a monitor for one of the suggested use cases.
   - The metric monitor is configured based on the selected use case. You can update the configuration to further customize it. See the [Metric monitor documentation](https://docs.datadoghq.com/monitors/types/metric.md) for more information.

## Set up a pipeline with the API{% #set-up-a-pipeline-with-the-api %}

1. Use the Observability Pipelines API to [create a pipeline](https://docs.datadoghq.com/api/latest/observability-pipelines.md#create-a-new-pipeline). See the API reference for example request payloads.

1. After creating the pipeline, [install the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=api&platform=docker&secrets_source=secrets_management) to send data through the pipeline.

   - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for the list of environment variables you need for the different sources, processor, and destinations when you install the Worker.

Use the [update a pipeline](https://docs.datadoghq.com/api/latest/observability-pipelines.md#update-a-pipeline) endpoint to make any changes to an existing pipeline.

See [Advanced Worker Configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md) for bootstrapping options.

## Set up a pipeline with Terraform{% #set-up-a-pipeline-with-terraform %}

{% alert level="warning" %}
[Terraform 3.84.0](https://github.com/DataDog/terraform-provider-datadog/releases/tag/v3.84.0) replaces standalone processors with [processor groups](https://docs.datadoghq.com/observability_pipelines/processors.md#processor-groups) and is a breaking change. If you want to upgrade to Terraform 3.84.0, see the [PR description](https://github.com/DataDog/terraform-provider-datadog/pull/3346) for instructions on how to migrate your existing resources.
{% /alert %}

1. You can use the [datadog_observability_pipeline](https://registry.terraform.io/providers/DataDog/datadog/latest/docs) module to create a pipeline using Terraform.

1. After creating the pipeline, [install the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=terraform&platform=docker&secrets_source=secrets_management) to send data through the pipeline.

   - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for the list of environment variables you need for the different sources, processor, and destinations when you install the Worker.

Use the [datadog_observability_pipeline](https://registry.terraform.io/providers/DataDog/datadog/latest/docs) module to make any changes to an existing pipeline.

See [Advanced Worker Configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md) for bootstrapping options.

## Clone a pipeline{% #clone-a-pipeline %}

To clone a pipeline in the UI:

1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines).
1. Select the pipeline you want to clone.
1. Click the cog at the top right side of the page, then select Clone Pipeline.

## Search for a component{% #search-for-a-component %}

To find a source, processor, or destination in a pipeline:

1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines).
1. Select a pipeline.
1. Click the cog at the top right side of the page, then select Search Components.
1. Enter the component's ID or type.

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/setup/search_components.d4e27775c9dd21850c80512d84442378.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/setup/search_components.d4e27775c9dd21850c80512d84442378.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The component search box showing sources and processors with their component IDs." /%}

## Delete a pipeline{% #delete-a-pipeline %}

To delete a pipeline in the UI:

1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines).
1. Select the pipeline you want to delete.
1. Click the cog at the top right side of the page, then select Delete Pipeline.

**Note**: You cannot delete an active pipeline. You must stop all Workers for a pipeline before you can delete it.

## Pipeline requirements and limits{% #pipeline-requirements-and-limits %}

- A pipeline must have at least one destination. If a processor group only has one destination, that destination cannot be deleted.
- You can add a total of 10 sources for a pipeline.
- You can add a total of 20 destinations for a pipeline.

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Use Pipeline Simulation to add, update, and validate your processors](https://docs.datadoghq.com/observability_pipelines/configuration/pipeline_simulation.md)
- [Update an existing pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md)
- [Advanced Worker configurations for Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md)
- [Run multiple pipelines on a host](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/run_multiple_pipelines_on_a_host.md)
- [Troubleshooting Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/monitoring_and_troubleshooting/troubleshooting.md)