Set up your pipelines and its sources, processors, and destinations in the Observability Pipelines UI.

1. Navigate to Observability Pipelines.
2. Select a template based on your use case.
   • Archive Logs
   • Dual Ship Logs
   • Generate Metrics
   • Log Enrichment
   • Log Volume Control
   • Sensitive Data Redaction
   • Split Logs
3. Select and set up your source.
4. Add processors to transform, redact, and enrich log data.
   • If you want to copy a processor, click the copy icon for that processor and then use command-v to paste it.
5. Select and set up destinations for your processed logs.

Add or remove components

Add another processor group

If you want to add another group of processors for a destination:

1. Click the plus sign (+) at the bottom of the existing processor group.
2. Click the name of the processor group to update it.
3. Optionally, enter a group filter. See Filter Syntax for more information.
4. Click Add to add processors to the group.
5. If you want to copy all processors in a group and paste them into the same processor group or a different group:
   1. Click the three dots on the processor group.
   2. Select Copy all processors.
   3. Select the desired processor group, and then paste the processors into it.
6. You can toggle the switch to enable and disable the processor group and also each individual processor.

Notes:
- Configuring a pipeline with processor groups is available for Worker versions 2.7 and later.
- There is a limit of 10 processor groups for a pipeline canvas.

Add another set of processors and destinations

If you want to add another set of processors and destinations, click the plus sign (+) to the left of the processor group to add another set of processors and destinations to the source.

To delete a processor group, you need to delete all destinations linked to that processor group. When the last destination is deleted, the processor group is removed with it.

Add another destination to a processor group

If you want to add an additional destination to a processor group, click the plus sign (+) to the right of the processor group.

To delete a destination, click on the pencil icon to the top right of the destination, and select Delete node.

Notes:

• If you delete a destination from a processor group that has multiple destinations, only the deleted destination is removed.
• If you delete a destination from a processor group that only has one destination, both the destination and the processor group are removed.

Install the Worker and deploy the pipeline

After you have set up your source, processors, and destinations:

1. Click Next: Install.
2. Select the platform on which you want to install the Worker.
3. Enter the environment variables for your sources and destinations, if applicable.
4. Follow the instructions on installing the Worker for your platform. The command provided in the UI to install the Worker has the relevant environment variables populated.
   • See Install the Worker for more information.
   • Note: If you are using a proxy, see the proxy option in Bootstrap options.
5. Enable out-of-the-box monitors for your pipeline.
   1. Navigate to the Pipelines page and find your pipeline.
   2. Click Enable monitors in the Monitors column for your pipeline.
   3. Click Start to set up a monitor for one of the suggested use cases.
      
      • The metric monitor is configured based on the selected use case. You can update the configuration to further customize it. See the Metric monitor documentation for more information.

After you have set up your pipeline, see Update Existing Pipelines if you want to make any changes to it.

1. You can use Observability Pipelines API to create a pipeline.

2. After creating the pipeline, install the Worker to send data through the pipeline.

   • See Environment Variables for the list of environment variables you need for the different sources, processor, and destinations when you install the Worker.

Note: Pipelines created using the API are read-only in the UI. Use the update a pipeline endpoint to make any changes to an existing pipeline.

1. You can use the datadog_observability_pipeline module to create a pipeline using Terraform.

2. After creating the pipeline, install the Worker to send data through the pipeline.

   • See Environment Variables for the list of environment variables you need for the different sources, processor, and destinations when you install the Worker.

Pipelines created using Terraform are read-only in the UI. Use the datadog_observability_pipeline module to make any changes to an existing pipeline.