Manage Observability Pipelines with the API or Terraform

Docs > Observability Pipelines > Observability Pipelines Guides > Manage Observability Pipelines with the API or Terraform

Overview

If you manage many Observability Pipelines deployments and want to reduce manual configuration errors, you can use the API or Terraform to programmatically manage your pipelines. This guide describes how to configure and update your pipelines with the API or Terraform.

Prerequisites

Before you begin, make sure you:

Have Datadog API and application keys for authentication.
Note: The API key must be enabled for Remote Configuration.
If you are going to use Terraform:
- Have the latest version of Terraform installed on your machine.
- Reviewed the Datadog Terraform provider and Observability Pipelines resource.
If you are going to use the API, review the Observability Pipelines API endpoint specifications and additional configuration parameters.

Manage pipelines with the API

You can perform CRUD (Create, Read, Update, Delete) operations with the Observability Pipelines API. This section describes how to use these endpoints in your workflow. For each example request, replace the following placeholders:

<PIPELINE_ID> with the identifier obtained when the pipeline was created
<DD_API_KEY> with your Datadog API key
<DD_APP_KEY> with your Datadog application key

The example payloads also include sample id values (such as my-processor-group and datadog-agent-source) for sources, processors, and destinations. These are names you choose and can rename to fit your own conventions. The type values (such as datadog_agent, filter, and datadog_logs) are fixed and must match a supported component type.

Create a pipeline

To create a pipeline, send a POST request with a JSON payload that defines the pipeline’s name and its main components: sources, processors, and destinations.

Example request:

curl -X POST "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>" \
-d '{
  "data": {
    "attributes": {
      "config": {
        "destinations": [
          { "id": "datadog-logs-destination", "type": "datadog_logs", "inputs": ["my-processor-group"] }
        ],
        "pipeline_type": "logs",
        "processor_groups": [
          {
            "enabled": true,
            "id": "my-processor-group",
            "include": "service:my-service",
            "inputs": [
              "datadog-agent-source"
            ],
            "processors": [
              { "id": "filter-processor", "enabled": true, "type": "filter", "include": "service:my-service" }
            ]
          }
        ],
        "sources": [
          { "id": "datadog-agent-source", "type": "datadog_agent" }
        ]
      },
      "name": "Main Observability Pipeline"
    },
    "type": "pipelines"
  }
}'

Retrieve a pipeline configuration

To audit or verify an existing pipeline configuration, send a GET request with the specific pipeline ID.

Example request:

curl -X GET "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines/<PIPELINE_ID>" \
-H "Accept: application/json" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>"

Update an existing pipeline

To update an existing pipeline’s configuration, send a PUT request with the pipeline changes in the JSON payload.

Example request:

curl -X PUT "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines/<PIPELINE_ID>" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>" \
-d '{
  "data": {
    "attributes": {
      "name": "Updated Pipeline Name",
      "config": {
        "sources": [
          { "id": "datadog-agent-source", "type": "datadog_agent" }
        ],
        "processors": [
          { "id": "filter-processor", "type": "filter", "include": "service:my-updated-service", "inputs": ["datadog-agent-source"] }
        ],
        "destinations": [
          { "id": "updated-datadog-logs-destination", "type": "datadog_logs", "inputs": ["filter-processor"] }
        ]
      }
    },
    "type": "pipelines"
  }
}'

Delete a pipeline

To delete a pipeline, send a DELETE request to the corresponding endpoint. A successful deletion results in a 204 status code indicating that the pipeline has been removed.

Note: The delete operation is irreversible. Use this endpoint only when you’re certain that the pipeline is no longer needed.

Example request:

curl -X DELETE "https://api.datadoghq.com/api/v2/remote_config/products/obs_pipelines/pipelines/<PIPELINE_ID>" \
-H "DD-API-KEY: <DD_API_KEY>" \
-H "DD-APPLICATION-KEY: <DD_APP_KEY>"

Manage pipelines with Terraform

You can use Terraform resources to create and deploy a pipeline.

Create a pipeline using Terraform

Define a pipeline using the datadog_observability_pipeline resource. Maintain this file in your version control system to track changes.

Set the following environment variables before you run Terraform, so that credentials aren’t stored in your configuration file:

export DD_API_KEY=<DD_API_KEY>
export DD_APP_KEY=<DD_APP_KEY>
export DD_HOST=

Example Terraform pipeline configuration:

terraform {
  required_providers {
    datadog = {
      source = "DataDog/datadog"
      version = "~> 3.84"
    }
  }
}

provider "datadog" {}

resource "datadog_observability_pipeline" "main" {
  name = "Main Observability Pipeline"

  config {
    source {
      id = "datadog-agent-source"

      datadog_agent {}
    }

    processor_group {
      id      = "filter-processor"
      enabled = true
      include = "service:my-service"
      inputs  = ["datadog-agent-source"]

      processor {
        id      = "filter-1"
        enabled = true
        include = "service:my-service"

        filter {}
      }
    }

    destination {
      id     = "datadog-logs-destination"
      inputs = ["filter-processor"]

      datadog_logs {}
    }
  }
}

Replace service:my-service with a search query that matches the logs you want the pipeline to process.

Deploy a pipeline with Terraform

After you define a new pipeline configuration or update an existing configuration, run the following Terraform commands to deploy your pipeline configuration:

terraform init
terraform plan
terraform apply

terraform init initializes your working directory.
terraform plan previews the changes being made.
terraform apply applies the changes, which creates or updates your pipeline accordingly.

After you deploy the configuration, install the Worker to send data through the pipeline. A pipeline doesn’t process data until at least one Worker is running for it.

Note: You cannot delete an active pipeline. Stop all Workers for the pipeline before you remove its resource block. See Delete a pipeline for more information.