Logs Pipelines
Security Monitoring is now available Security Monitoring is now available

Logs Pipelines

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Create a pipeline

post https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines

Overview

Create a pipeline in your organization.

Request

Body Data (required)

Definition of the new pipeline.

Expand All

Field

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object]

Ordered list of processors in this pipeline.

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "is_enabled": false,
  "name": "string",
  "processors": []
}

Response

OK

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Expand All

Field

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object]

Ordered list of processors in this pipeline.

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "string",
  "processors": [],
  "type": "pipeline"
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Field

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Field

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example


                                        # Curl command
curl -X post https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_CLIENT_APP_KEY}" \
-d @- << EOF
{
  "name": null
}
EOF

                                        

Delete a pipeline

delete https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines/{pipeline_id}

Overview

Delete a given pipeline from your organization. This endpoint takes no JSON arguments.

Arguments

Path Parameters

Name

Type

Description

pipeline_id [required]

string

ID of the pipeline to delete.

Response

OK

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Field

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Field

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example


                                        # Path parameters
export pipeline_id="CHANGE_ME"
# Curl command curl -X delete https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines/${pipeline_id} \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_CLIENT_APP_KEY}"

Get a pipeline

get https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines/{pipeline_id}

Overview

Get a specific pipeline from your organization. This endpoint takes no JSON arguments.

Arguments

Path Parameters

Name

Type

Description

pipeline_id [required]

string

ID of the pipeline to get.

Response

OK

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Expand All

Field

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object]

Ordered list of processors in this pipeline.

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "string",
  "processors": [],
  "type": "pipeline"
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Field

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Field

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example


                                        # Path parameters
export pipeline_id="CHANGE_ME"
# Curl command curl -X get https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines/${pipeline_id} \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_CLIENT_APP_KEY}"

Get all pipelines

get https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines

Overview

Get all pipelines from your organization. This endpoint takes no JSON arguments.

Response

OK

Array of pipeline ID strings.

Expand All

Field

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name

string

Name of the pipeline.

processors

[object]

Ordered list of processors in this pipeline.

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "string",
  "processors": [],
  "type": "pipeline"
}

Forbidden

Error response object.

Expand All

Field

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example


                                        # Curl command
curl -X get https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_CLIENT_APP_KEY}"

                                        

Get pipeline order

get https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipeline-order

Overview

Get the current order of your pipelines. This endpoint takes no JSON arguments.

Response

OK

Object containing the ordered list of pipeline IDs.

Expand All

Field

Type

Description

pipeline_ids [required]

[string]

Ordered Array of <PIPELINE_ID> strings, the order of pipeline IDs in the array define the overall Pipelines order for Datadog.

{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}

Forbidden

Error response object.

Expand All

Field

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example


                                        # Curl command
curl -X get https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipeline-order \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_CLIENT_APP_KEY}"

                                        

Update a pipeline

put https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines/{pipeline_id}

Overview

Update a given pipeline configuration to change it’s processors or their order.

Note: Using this method updates your pipeline configuration by replacing your current configuration with the new one sent to your Datadog organization.

Arguments

Path Parameters

Name

Type

Description

pipeline_id [required]

string

ID of the pipeline to delete.

Request

Body Data (required)

New definition of the pipeline.

Expand All

Field

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object]

Ordered list of processors in this pipeline.

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "is_enabled": false,
  "name": "string",
  "processors": []
}

Response

OK

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Expand All

Field

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object]

Ordered list of processors in this pipeline.

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "string",
  "processors": [],
  "type": "pipeline"
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Field

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Field

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example


                                        # Path parameters
export pipeline_id="CHANGE_ME"
# Curl command curl -X put https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipelines/${pipeline_id} \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_CLIENT_APP_KEY}" \ -d @- << EOF { "name": null } EOF

Update pipeline order

put https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipeline-order

Overview

Update the order of your pipelines. Since logs are processed sequentially, reordering a pipeline may change the structure and content of the data processed by other pipelines and their processors.

Note: Using the PUT method updates your pipeline order by replacing your current order with the new one sent to your Datadog organization.

Request

Body Data (required)

Object containing the new ordered list of pipeline IDs.

Expand All

Field

Type

Description

pipeline_ids [required]

[string]

Ordered Array of <PIPELINE_ID> strings, the order of pipeline IDs in the array define the overall Pipelines order for Datadog.

{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}

Response

OK

Object containing the ordered list of pipeline IDs.

Expand All

Field

Type

Description

pipeline_ids [required]

[string]

Ordered Array of <PIPELINE_ID> strings, the order of pipeline IDs in the array define the overall Pipelines order for Datadog.

{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Field

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Field

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Unprocessable Entity

Response returned by the Logs API when errors occur.

Expand All

Field

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Code Example


                                        # Curl command
curl -X put https://api.datadoghq.comhttps://api.datadoghq.eu/api/v1/logs/config/pipeline-order \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_CLIENT_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_CLIENT_APP_KEY}" \
-d @- << EOF
{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}
EOF