---
title: Manage Observability Pipelines with the API or Terraform
description: Learn how to create and update pipelines using the API or Terraform.
breadcrumbs: >-
  Docs > Observability Pipelines > Observability Pipelines Guides > Manage
  Observability Pipelines with the API or Terraform
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Manage Observability Pipelines with the API or Terraform

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

If you manage many Observability Pipelines deployments and want to reduce manual configuration errors, you can use the API or Terraform to programmatically manage your pipelines. This guide describes how to configure and update your pipelines with the API or Terraform.

## Prerequisites{% #prerequisites %}

Before you begin, make sure you:

- Have Datadog API and application keys for authentication.**Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup).
- If you are going to use Terraform:
  - Have the latest version of Terraform installed on your machine.
  - Reviewed the [Datadog Terraform provider](https://registry.terraform.io/providers/DataDog/datadog/latest/docs) and [Observability Pipelines resource](https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/observability_pipeline).
- If you are going to use the API, review the [Observability Pipelines API](https://docs.datadoghq.com/api/latest/observability-pipelines.md) endpoint specifications and additional configuration parameters.

## Manage pipelines with the API{% #manage-pipelines-with-the-api %}

You can perform CRUD (Create, Read, Update, Delete) operations with the Observability Pipelines API. This section describes how to use these endpoints in your workflow. For each example request, replace the following placeholders:

- `<PIPELINE_ID>` with the identifier obtained when the pipeline was created
- `<DD_API_KEY>` with your Datadog API key
- `<DD_APP_KEY>` with your Datadog application key

The example payloads also include sample `id` values (such as `my-processor-group` and `datadog-agent-source`) for sources, processors, and destinations. These are names you choose and can rename to fit your own conventions. The `type` values (such as `datadog_agent`, `filter`, and `datadog_logs`) are fixed and must match a supported component type.

### Create a pipeline{% #create-a-pipeline %}

To [create a pipeline](https://docs.datadoghq.com/api/latest/observability-pipelines/create-a-new-pipeline.md), send a `POST` request with a JSON payload that defines the pipeline's name and its main components: sources, processors, and destinations.

Example request:

```bash
curl -X POST "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>" \
-d '{
  "data": {
    "attributes": {
      "config": {
        "destinations": [
          { "id": "datadog-logs-destination", "type": "datadog_logs", "inputs": ["my-processor-group"] }
        ],
        "pipeline_type": "logs",
        "processor_groups": [
          {
            "enabled": true,
            "id": "my-processor-group",
            "include": "service:my-service",
            "inputs": [
              "datadog-agent-source"
            ],
            "processors": [
              { "id": "filter-processor", "enabled": true, "type": "filter", "include": "service:my-service" }
            ]
          }
        ],
        "sources": [
          { "id": "datadog-agent-source", "type": "datadog_agent" }
        ]
      },
      "name": "Main Observability Pipeline"
    },
    "type": "pipelines"
  }
}'
```

### Retrieve a pipeline configuration{% #retrieve-a-pipeline-configuration %}

To [audit or verify an existing pipeline configuration](https://docs.datadoghq.com/api/latest/observability-pipelines/get-a-specific-pipeline.md), send a `GET` request with the specific pipeline ID.

Example request:

```bash
curl -X GET "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines/<PIPELINE_ID>" \
-H "Accept: application/json" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>"
```

### Update an existing pipeline{% #update-an-existing-pipeline %}

To [update an existing pipeline's configuration](https://docs.datadoghq.com/api/latest/observability-pipelines/update-a-pipeline.md), send a `PUT` request with the pipeline changes in the JSON payload.

Example request:

```bash
curl -X PUT "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines/<PIPELINE_ID>" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>" \
-d '{
  "data": {
    "attributes": {
      "name": "Updated Pipeline Name",
      "config": {
        "sources": [
          { "id": "datadog-agent-source", "type": "datadog_agent" }
        ],
        "processors": [
          { "id": "filter-processor", "type": "filter", "include": "service:my-updated-service", "inputs": ["datadog-agent-source"] }
        ],
        "destinations": [
          { "id": "updated-datadog-logs-destination", "type": "datadog_logs", "inputs": ["filter-processor"] }
        ]
      }
    },
    "type": "pipelines"
  }
}'
```

### Delete a pipeline{% #delete-a-pipeline %}

To [delete a pipeline](https://docs.datadoghq.com/api/latest/observability-pipelines/delete-a-pipeline.md), send a `DELETE` request to the corresponding endpoint. A successful deletion results in a `204` status code indicating that the pipeline has been removed.

**Note**: The delete operation is irreversible. Use this endpoint only when you're certain that the pipeline is no longer needed.

Example request:

```bash
curl -X DELETE "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines/<PIPELINE_ID>" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>"
```

## Manage pipelines with Terraform{% #manage-pipelines-with-terraform %}

You can use Terraform resources to create and deploy a pipeline.

### Create a pipeline using Terraform{% #create-a-pipeline-using-terraform %}

Define a pipeline using the [datadog_observability_pipeline](https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/observability_pipeline) resource. Maintain this file in your version control system to track changes.

Set the following environment variables before you run Terraform, so that credentials aren't stored in your configuration file:

```shell
export DD_API_KEY=<DD_API_KEY>
export DD_APP_KEY=<DD_APP_KEY>
export DD_HOST=<YOUR_DATADOG_API_URL>
```

Example Terraform pipeline configuration:

```hcl
terraform {
  required_providers {
    datadog = {
      source = "DataDog/datadog"
      version = "~> 3.84"
    }
  }
}

provider "datadog" {}

resource "datadog_observability_pipeline" "main" {
  name = "Main Observability Pipeline"

  config {
    source {
      id = "datadog-agent-source"

      datadog_agent {}
    }

    processor_group {
      id      = "filter-processor"
      enabled = true
      include = "service:my-service"
      inputs  = ["datadog-agent-source"]

      processor {
        id      = "filter-1"
        enabled = true
        include = "service:my-service"

        filter {}
      }
    }

    destination {
      id     = "datadog-logs-destination"
      inputs = ["filter-processor"]

      datadog_logs {}
    }
  }
}
```

Replace `service:my-service` with a search query that matches the logs you want the pipeline to process.

### Deploy a pipeline with Terraform{% #deploy-a-pipeline-with-terraform %}

After you define a new pipeline configuration or update an existing configuration, run the following Terraform commands to deploy your pipeline configuration:

```bash
terraform init
terraform plan
terraform apply
```

- `terraform init` initializes your working directory.
- `terraform plan` previews the changes being made.
- `terraform apply` applies the changes, which creates or updates your pipeline accordingly.

After you deploy the configuration, [install the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md?tab=logs#set-up-a-pipeline-with-the-api) to send data through the pipeline. A pipeline doesn't process data until at least one Worker is running for it.

**Note**: You cannot delete an active pipeline. Stop all Workers for the pipeline before you remove its resource block. See [Delete a pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md?tab=logs#delete-a-pipeline) for more information.

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Set up a pipeline](https://docs.datadoghq.com/observability_pipelines/set_up_pipelines.md)
- [Observability Pipelines API](https://docs.datadoghq.com/api/latest/observability-pipelines.md)