--- title: Observability Pipelines description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > API Reference > Observability Pipelines --- # Observability Pipelines Observability Pipelines allows you to collect and process logs within your own infrastructure, and then route them to downstream integrations. ## List pipelines{% #list-pipelines %} {% tab title="v2" %} | Datadog site | API endpoint | | ----------------- | ---------------------------------------------------------------- | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/obs-pipelines/pipelines | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/obs-pipelines/pipelines | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/obs-pipelines/pipelines | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/obs-pipelines/pipelines | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/obs-pipelines/pipelines | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/obs-pipelines/pipelines | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines | ### Overview Retrieve a list of pipelines. This endpoint requires the `observability_pipelines_read` permission. ### Arguments #### Query Strings | Name | Type | Description | | ------------ | ------- | -------------------------------------------------------- | | page[size] | integer | Size for a given page. The maximum allowed value is 100. | | page[number] | integer | Specific page number to return. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Represents the response payload containing a list of pipelines and associated metadata. | Parent field | Field | Type | Description | | -------------------- | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | [object] | The `schema` `data`. | | data | attributes [*required*] | object | Defines the pipeline's name and its components (sources, processors, and destinations). | | attributes | config [*required*] | object | Specifies the pipeline's configuration, including its sources, processors, and destinations. | | config | destinations [*required*] | [ ] | A list of destination components where processed logs are sent. | | destinations | Option 1 | object | The `http_client` destination sends data to an HTTP endpoint. | **Supported pipeline types:** logs, metrics | | Option 1 | auth_strategy | enum | HTTP authentication strategy. Allowed enum values: `none,basic,bearer` | | Option 1 | compression | object | Compression configuration for HTTP requests. | | compression | algorithm [*required*] | enum | Compression algorithm. Allowed enum values: `gzip` | | Option 1 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 1 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json` | | Option 1 | id [*required*] | string | The unique identifier for this component. | | Option 1 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 1 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 1 | type [*required*] | enum | The destination type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 1 | uri_key | string | Name of the environment variable or secret that holds the HTTP endpoint URI. | | Option 1 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | destinations | Option 2 | object | The `amazon_opensearch` destination writes logs to Amazon OpenSearch. | **Supported pipeline types:** logs | | Option 2 | auth [*required*] | object | Authentication settings for the Amazon OpenSearch destination. The `strategy` field determines whether basic or AWS-based authentication is used. | | auth | assume_role | string | The ARN of the role to assume (used with `aws` strategy). | | auth | aws_region | string | AWS region | | auth | external_id | string | External ID for the assumed role (used with `aws` strategy). | | auth | session_name | string | Session name for the assumed role (used with `aws` strategy). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | Option 2 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 2 | bulk_index | string | The index to write logs to. | | Option 2 | id [*required*] | string | The unique identifier for this component. | | Option 2 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 2 | type [*required*] | enum | The destination type. The value should always be `amazon_opensearch`. Allowed enum values: `amazon_opensearch` | | destinations | Option 3 | object | The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | bucket [*required*] | string | S3 bucket name. | | Option 3 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 3 | id [*required*] | string | Unique identifier for the destination component. | | Option 3 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 3 | key_prefix | string | Optional prefix for object keys. | | Option 3 | region [*required*] | string | AWS region of the S3 bucket. | | Option 3 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The destination type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | destinations | Option 4 | object | The `amazon_s3_generic` destination sends your logs to an Amazon S3 bucket. | **Supported pipeline types:** logs | | Option 4 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 4 | batch_settings | object | Event batching settings | | batch_settings | batch_size | int64 | Maximum batch size in bytes. | | batch_settings | timeout_secs | int64 | Maximum number of seconds to wait before flushing the batch. | | Option 4 | bucket [*required*] | string | S3 bucket name. | | Option 4 | compression [*required*] | | Compression algorithm applied to encoded logs. | | compression | Option 1 | object | Zstd compression. | | Option 1 | algorithm [*required*] | enum | The compression type. Always `zstd`. Allowed enum values: `zstd` | | Option 1 | level [*required*] | int64 | Zstd compression level. | | compression | Option 2 | object | Gzip compression. | | Option 2 | algorithm [*required*] | enum | The compression type. Always `gzip`. Allowed enum values: `gzip` | | Option 2 | level [*required*] | int64 | Gzip compression level. | | compression | Option 3 | object | Snappy compression. | | Option 3 | algorithm [*required*] | enum | The compression type. Always `snappy`. Allowed enum values: `snappy` | | Option 4 | encoding [*required*] | | Encoding format for the destination. | | encoding | Option 1 | object | JSON encoding. | | Option 1 | type [*required*] | enum | The encoding type. Always `json`. Allowed enum values: `json` | | encoding | Option 2 | object | Parquet encoding. | | Option 2 | type [*required*] | enum | The encoding type. Always `parquet`. Allowed enum values: `parquet` | | Option 4 | id [*required*] | string | Unique identifier for the destination component. | | Option 4 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 4 | key_prefix | string | Optional prefix for object keys. | | Option 4 | region [*required*] | string | AWS region of the S3 bucket. | | Option 4 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 4 | type [*required*] | enum | The destination type. Always `amazon_s3_generic`. Allowed enum values: `amazon_s3_generic` | | destinations | Option 5 | object | The `amazon_security_lake` destination sends your logs to Amazon Security Lake. | **Supported pipeline types:** logs | | Option 5 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 5 | bucket [*required*] | string | Name of the Amazon S3 bucket in Security Lake (3-63 characters). | | Option 5 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 5 | custom_source_name [*required*] | string | Custom source name for the logs in Security Lake. | | Option 5 | id [*required*] | string | Unique identifier for the destination component. | | Option 5 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 5 | region [*required*] | string | AWS region of the S3 bucket. | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The destination type. Always `amazon_security_lake`. Allowed enum values: `amazon_security_lake` | | destinations | Option 6 | object | The `azure_storage` destination forwards logs to an Azure Blob Storage container. | **Supported pipeline types:** logs | | Option 6 | blob_prefix | string | Optional prefix for blobs written to the container. | | Option 6 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 6 | connection_string_key | string | Name of the environment variable or secret that holds the Azure Storage connection string. | | Option 6 | container_name [*required*] | string | The name of the Azure Blob Storage container to store logs in. | | Option 6 | id [*required*] | string | The unique identifier for this component. | | Option 6 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 6 | type [*required*] | enum | The destination type. The value should always be `azure_storage`. Allowed enum values: `azure_storage` | | destinations | Option 7 | object | The `cloud_prem` destination sends logs to Datadog CloudPrem. | **Supported pipeline types:** logs | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the CloudPrem endpoint URL. | | Option 7 | id [*required*] | string | The unique identifier for this component. | | Option 7 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 7 | type [*required*] | enum | The destination type. The value should always be `cloud_prem`. Allowed enum values: `cloud_prem` | | destinations | Option 8 | object | The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM. | **Supported pipeline types:** logs | | Option 8 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 8 | compression | object | Compression configuration for log events. | | compression | algorithm [*required*] | enum | Compression algorithm for log events. Allowed enum values: `gzip,zlib` | | compression | level | int64 | Compression level. | | Option 8 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 8 | endpoint_url_key | string | Name of the environment variable or secret that holds the CrowdStrike endpoint URL. | | Option 8 | id [*required*] | string | The unique identifier for this component. | | Option 8 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | token_key | string | Name of the environment variable or secret that holds the CrowdStrike API token. | | Option 8 | type [*required*] | enum | The destination type. The value should always be `crowdstrike_next_gen_siem`. Allowed enum values: `crowdstrike_next_gen_siem` | | destinations | Option 9 | object | The `datadog_logs` destination forwards logs to Datadog Log Management. | **Supported pipeline types:** logs | | Option 9 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 9 | id [*required*] | string | The unique identifier for this component. | | Option 9 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 9 | routes | [object] | A list of routing rules that forward matching logs to Datadog using dedicated API keys. | | routes | api_key_key | string | Name of the environment variable or secret that stores the Datadog API key used by this route. | | routes | include | string | A Datadog search query that determines which logs are forwarded using this route. | | routes | route_id | string | Unique identifier for this route within the destination. | | routes | site | string | Datadog site where matching logs are sent (for example, `us1`). | | Option 9 | type [*required*] | enum | The destination type. The value should always be `datadog_logs`. Allowed enum values: `datadog_logs` | | destinations | Option 10 | object | The `elasticsearch` destination writes logs to an Elasticsearch cluster. | **Supported pipeline types:** logs | | Option 10 | api_version | enum | The Elasticsearch API version to use. Set to `auto` to auto-detect. Allowed enum values: `auto,v6,v7,v8` | | Option 10 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 10 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 10 | bulk_index | string | The index to write logs to in Elasticsearch. | | Option 10 | data_stream | object | Configuration options for writing to Elasticsearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 10 | endpoint_url_key | string | Name of the environment variable or secret that holds the Elasticsearch endpoint URL. | | Option 10 | id [*required*] | string | The unique identifier for this component. | | Option 10 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 10 | type [*required*] | enum | The destination type. The value should always be `elasticsearch`. Allowed enum values: `elasticsearch` | | destinations | Option 11 | object | The `google_chronicle` destination sends logs to Google Chronicle. | **Supported pipeline types:** logs | | Option 11 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 11 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 11 | customer_id [*required*] | string | The Google Chronicle customer ID. | | Option 11 | encoding | enum | The encoding format for the logs sent to Chronicle. Allowed enum values: `json,raw_message` | | Option 11 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Chronicle endpoint URL. | | Option 11 | id [*required*] | string | The unique identifier for this component. | | Option 11 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 11 | log_type | string | The log type metadata associated with the Chronicle destination. | | Option 11 | type [*required*] | enum | The destination type. The value should always be `google_chronicle`. Allowed enum values: `google_chronicle` | | destinations | Option 12 | object | The `google_cloud_storage` destination stores logs in a Google Cloud Storage (GCS) bucket. It requires a bucket name, Google Cloud authentication, and metadata fields. | **Supported pipeline types:** logs | | Option 12 | acl | enum | Access control list setting for objects written to the bucket. Allowed enum values: `private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control` | | Option 12 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 12 | bucket [*required*] | string | Name of the GCS bucket. | | Option 12 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 12 | id [*required*] | string | Unique identifier for the destination component. | | Option 12 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 12 | key_prefix | string | Optional prefix for object keys within the GCS bucket. | | Option 12 | metadata | [object] | Custom metadata to attach to each object uploaded to the GCS bucket. | | metadata | name [*required*] | string | The metadata key. | | metadata | value [*required*] | string | The metadata value. | | Option 12 | storage_class [*required*] | enum | Storage class used for objects stored in GCS. Allowed enum values: `STANDARD,NEARLINE,COLDLINE,ARCHIVE` | | Option 12 | type [*required*] | enum | The destination type. Always `google_cloud_storage`. Allowed enum values: `google_cloud_storage` | | destinations | Option 13 | object | The `google_pubsub` destination publishes logs to a Google Cloud Pub/Sub topic. | **Supported pipeline types:** logs | | Option 13 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 13 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 13 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 13 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL. | | Option 13 | id [*required*] | string | The unique identifier for this component. | | Option 13 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 13 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub topic. | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | topic [*required*] | string | The Pub/Sub topic name to publish logs to. | | Option 13 | type [*required*] | enum | The destination type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | destinations | Option 14 | object | The `kafka` destination sends logs to Apache Kafka topics. | **Supported pipeline types:** logs | | Option 14 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 14 | compression | enum | Compression codec for Kafka messages. Allowed enum values: `none,gzip,snappy,lz4,zstd` | | Option 14 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 14 | headers_key | string | The field name to use for Kafka message headers. | | Option 14 | id [*required*] | string | The unique identifier for this component. | | Option 14 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 14 | key_field | string | The field name to use as the Kafka message key. | | Option 14 | librdkafka_options | [object] | Optional list of advanced Kafka producer configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 14 | message_timeout_ms | int64 | Maximum time in milliseconds to wait for message delivery confirmation. | | Option 14 | rate_limit_duration_secs | int64 | Duration in seconds for the rate limit window. | | Option 14 | rate_limit_num | int64 | Maximum number of messages allowed per rate limit duration. | | Option 14 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 14 | socket_timeout_ms | int64 | Socket timeout in milliseconds for network requests. | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | topic [*required*] | string | The Kafka topic name to publish logs to. | | Option 14 | type [*required*] | enum | The destination type. The value should always be `kafka`. Allowed enum values: `kafka` | | destinations | Option 15 | object | The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel. | **Supported pipeline types:** logs | | Option 15 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 15 | client_id [*required*] | string | Azure AD client ID used for authentication. | | Option 15 | client_secret_key | string | Name of the environment variable or secret that holds the Azure AD client secret. | | Option 15 | dce_uri_key | string | Name of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI. | | Option 15 | dcr_immutable_id [*required*] | string | The immutable ID of the Data Collection Rule (DCR). | | Option 15 | id [*required*] | string | The unique identifier for this component. | | Option 15 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 15 | table [*required*] | string | The name of the Log Analytics table where logs are sent. | | Option 15 | tenant_id [*required*] | string | Azure AD tenant ID. | | Option 15 | type [*required*] | enum | The destination type. The value should always be `microsoft_sentinel`. Allowed enum values: `microsoft_sentinel` | | destinations | Option 16 | object | The `new_relic` destination sends logs to the New Relic platform. | **Supported pipeline types:** logs | | Option 16 | account_id_key | string | Name of the environment variable or secret that holds the New Relic account ID. | | Option 16 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 16 | id [*required*] | string | The unique identifier for this component. | | Option 16 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 16 | license_key_key | string | Name of the environment variable or secret that holds the New Relic license key. | | Option 16 | region [*required*] | enum | The New Relic region. Allowed enum values: `us,eu` | | Option 16 | type [*required*] | enum | The destination type. The value should always be `new_relic`. Allowed enum values: `new_relic` | | destinations | Option 17 | object | The `opensearch` destination writes logs to an OpenSearch cluster. | **Supported pipeline types:** logs | | Option 17 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 17 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 17 | bulk_index | string | The index to write logs to. | | Option 17 | data_stream | object | Configuration options for writing to OpenSearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 17 | endpoint_url_key | string | Name of the environment variable or secret that holds the OpenSearch endpoint URL. | | Option 17 | id [*required*] | string | The unique identifier for this component. | | Option 17 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 17 | type [*required*] | enum | The destination type. The value should always be `opensearch`. Allowed enum values: `opensearch` | | destinations | Option 18 | object | The `rsyslog` destination forwards logs to an external `rsyslog` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 18 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 18 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog server endpoint URL. | | Option 18 | id [*required*] | string | The unique identifier for this component. | | Option 18 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 18 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 18 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 18 | type [*required*] | enum | The destination type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | destinations | Option 19 | object | The `sentinel_one` destination sends logs to SentinelOne. | **Supported pipeline types:** logs | | Option 19 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 19 | id [*required*] | string | The unique identifier for this component. | | Option 19 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 19 | region [*required*] | enum | The SentinelOne region to send logs to. Allowed enum values: `us,eu,ca,data_set_us` | | Option 19 | token_key | string | Name of the environment variable or secret that holds the SentinelOne API token. | | Option 19 | type [*required*] | enum | The destination type. The value should always be `sentinel_one`. Allowed enum values: `sentinel_one` | | destinations | Option 20 | object | The `socket` destination sends logs over TCP or UDP to a remote server. | **Supported pipeline types:** logs | | Option 20 | address_key | string | Name of the environment variable or secret that holds the socket address (host:port). | | Option 20 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 20 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 20 | framing [*required*] | | Framing method configuration. | | framing | Option 1 | object | Each log event is delimited by a newline character. | | Option 1 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod` object. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Event data is not delimited at all. | | Option 2 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod` object. Allowed enum values: `bytes` | | framing | Option 3 | object | Each log event is separated using the specified delimiter character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used as a delimiter. | | Option 3 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod` object. Allowed enum values: `character_delimited` | | Option 20 | id [*required*] | string | The unique identifier for this component. | | Option 20 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 20 | mode [*required*] | enum | Protocol used to send logs. Allowed enum values: `tcp,udp` | | Option 20 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 20 | type [*required*] | enum | The destination type. The value should always be `socket`. Allowed enum values: `socket` | | destinations | Option 21 | object | The `splunk_hec` destination forwards logs to Splunk using the HTTP Event Collector (HEC). | **Supported pipeline types:** logs | | Option 21 | auto_extract_timestamp | boolean | If `true`, Splunk tries to extract timestamps from incoming log events. If `false`, Splunk assigns the time the event was received. | | Option 21 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 21 | encoding | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 21 | endpoint_url_key | string | Name of the environment variable or secret that holds the Splunk HEC endpoint URL. | | Option 21 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 21 | index | string | Optional name of the Splunk index where logs are written. | | Option 21 | indexed_fields | [string] | List of log field names to send as indexed fields to Splunk HEC. Available only when `encoding` is `json`. | | Option 21 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 21 | sourcetype | string | The Splunk sourcetype to assign to log events. | | Option 21 | token_key | string | Name of the environment variable or secret that holds the Splunk HEC token. | | Option 21 | type [*required*] | enum | The destination type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | destinations | Option 22 | object | The `sumo_logic` destination forwards logs to Sumo Logic. | **Supported pipeline types:** logs | | Option 22 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 22 | encoding | enum | The output encoding format. Allowed enum values: `json,raw_message,logfmt` | | Option 22 | endpoint_url_key | string | Name of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL. | | Option 22 | header_custom_fields | [object] | A list of custom headers to include in the request to Sumo Logic. | | header_custom_fields | name [*required*] | string | The header field name. | | header_custom_fields | value [*required*] | string | The header field value. | | Option 22 | header_host_name | string | Optional override for the host name header. | | Option 22 | header_source_category | string | Optional override for the source category header. | | Option 22 | header_source_name | string | Optional override for the source name header. | | Option 22 | id [*required*] | string | The unique identifier for this component. | | Option 22 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 22 | type [*required*] | enum | The destination type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | destinations | Option 23 | object | The `syslog_ng` destination forwards logs to an external `syslog-ng` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 23 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 23 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog-ng server endpoint URL. | | Option 23 | id [*required*] | string | The unique identifier for this component. | | Option 23 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 23 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 23 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 23 | type [*required*] | enum | The destination type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | destinations | Option 24 | object | The `datadog_metrics` destination forwards metrics to Datadog. | **Supported pipeline types:** metrics | | Option 24 | id [*required*] | string | The unique identifier for this component. | | Option 24 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 24 | type [*required*] | enum | The destination type. The value should always be `datadog_metrics`. Allowed enum values: `datadog_metrics` | | config | pipeline_type | enum | The type of data being ingested. Defaults to `logs` if not specified. Allowed enum values: `logs,metrics` | | config | processor_groups | [object] | A list of processor groups that transform or enrich log data. | | processor_groups | display_name | string | The display name for a component. | | processor_groups | enabled [*required*] | boolean | Whether this processor group is enabled. | | processor_groups | id [*required*] | string | The unique identifier for the processor group. | | processor_groups | include [*required*] | string | Conditional expression for when this processor group should execute. | | processor_groups | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processor_groups | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | processors | [object] | **DEPRECATED**: A list of processor groups that transform or enrich log data. | **Deprecated:** This field is deprecated, you should now use the processor_groups field. | | processors | display_name | string | The display name for a component. | | processors | enabled [*required*] | boolean | Whether this processor group is enabled. | | processors | id [*required*] | string | The unique identifier for the processor group. | | processors | include [*required*] | string | Conditional expression for when this processor group should execute. | | processors | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processors | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | sources [*required*] | [ ] | A list of configured data sources for the pipeline. | | sources | Option 1 | object | The `datadog_agent` source collects logs/metrics from the Datadog Agent. | **Supported pipeline types:** logs, metrics | | Option 1 | address_key | string | Name of the environment variable or secret that holds the listen address for the Datadog Agent source. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | type [*required*] | enum | The source type. The value should always be `datadog_agent`. Allowed enum values: `datadog_agent` | | sources | Option 2 | object | The `amazon_data_firehose` source ingests logs from AWS Data Firehose. | **Supported pipeline types:** logs | | Option 2 | address_key | string | Name of the environment variable or secret that holds the Firehose delivery stream address. | | Option 2 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 2 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 2 | type [*required*] | enum | The source type. The value should always be `amazon_data_firehose`. Allowed enum values: `amazon_data_firehose` | | sources | Option 3 | object | The `amazon_s3` source ingests logs from an Amazon S3 bucket. It supports AWS authentication and TLS encryption. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | region [*required*] | string | AWS region where the S3 bucket resides. | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The source type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | Option 3 | url_key | string | Name of the environment variable or secret that holds the S3 bucket URL. | | sources | Option 4 | object | The `fluent_bit` source ingests logs from Fluent Bit. | **Supported pipeline types:** logs | | Option 4 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent Bit receiver. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 4 | type [*required*] | enum | The source type. The value should always be `fluent_bit`. Allowed enum values: `fluent_bit` | | sources | Option 5 | object | The `fluentd` source ingests logs from a Fluentd-compatible service. | **Supported pipeline types:** logs | | Option 5 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent receiver. | | Option 5 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The source type. The value should always be `fluentd. Allowed enum values: `fluentd` | | sources | Option 6 | object | The `google_pubsub` source ingests logs from a Google Cloud Pub/Sub subscription. | **Supported pipeline types:** logs | | Option 6 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 6 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub subscription. | | Option 6 | subscription [*required*] | string | The Pub/Sub subscription name from which messages are consumed. | | Option 6 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 6 | type [*required*] | enum | The source type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | sources | Option 7 | object | The `http_client` source scrapes logs from HTTP endpoints at regular intervals. | **Supported pipeline types:** logs | | Option 7 | auth_strategy | enum | Optional authentication strategy for HTTP requests. Allowed enum values: `none,basic,bearer,custom` | | Option 7 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 7 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the HTTP endpoint URL to scrape. | | Option 7 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 7 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 7 | scrape_interval_secs | int64 | The interval (in seconds) between HTTP scrape requests. | | Option 7 | scrape_timeout_secs | int64 | The timeout (in seconds) for each scrape request. | | Option 7 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 7 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 7 | type [*required*] | enum | The source type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 7 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | sources | Option 8 | object | The `http_server` source collects logs over HTTP POST from external services. | **Supported pipeline types:** logs | | Option 8 | address_key | string | Name of the environment variable or secret that holds the listen address for the HTTP server. | | Option 8 | auth_strategy [*required*] | enum | HTTP authentication method. Allowed enum values: `none,plain` | | Option 8 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 8 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 8 | id [*required*] | string | Unique ID for the HTTP server source. | | Option 8 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `plain`). | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | type [*required*] | enum | The source type. The value should always be `http_server`. Allowed enum values: `http_server` | | Option 8 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `plain`). | | sources | Option 9 | object | The `kafka` source ingests data from Apache Kafka topics. | **Supported pipeline types:** logs | | Option 9 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 9 | group_id [*required*] | string | Consumer group ID used by the Kafka client. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 9 | librdkafka_options | [object] | Optional list of advanced Kafka client configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 9 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 9 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 9 | topics [*required*] | [string] | A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified. | | Option 9 | type [*required*] | enum | The source type. The value should always be `kafka`. Allowed enum values: `kafka` | | sources | Option 10 | object | The `logstash` source ingests logs from a Logstash forwarder. | **Supported pipeline types:** logs | | Option 10 | address_key | string | Name of the environment variable or secret that holds the listen address for the Logstash receiver. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 10 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 10 | type [*required*] | enum | The source type. The value should always be `logstash`. Allowed enum values: `logstash` | | sources | Option 11 | object | The `rsyslog` source listens for logs over TCP or UDP from an `rsyslog` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 11 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog receiver. | | Option 11 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 11 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 11 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 11 | type [*required*] | enum | The source type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | sources | Option 12 | object | The `socket` source ingests logs over TCP or UDP. | **Supported pipeline types:** logs | | Option 12 | address_key | string | Name of the environment variable or secret that holds the listen address for the socket. | | Option 12 | framing [*required*] | | Framing method configuration for the socket source. | | framing | Option 1 | object | Byte frames which are delimited by a newline character. | | Option 1 | method [*required*] | enum | Byte frames which are delimited by a newline character. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). | | Option 2 | method [*required*] | enum | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). Allowed enum values: `bytes` | | framing | Option 3 | object | Byte frames which are delimited by a chosen character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used to delimit events. | | Option 3 | method [*required*] | enum | Byte frames which are delimited by a chosen character. Allowed enum values: `character_delimited` | | framing | Option 4 | object | Byte frames according to the octet counting format as per RFC6587. | | Option 4 | method [*required*] | enum | Byte frames according to the octet counting format as per RFC6587. Allowed enum values: `octet_counting` | | framing | Option 5 | object | Byte frames which are chunked GELF messages. | | Option 5 | method [*required*] | enum | Byte frames which are chunked GELF messages. Allowed enum values: `chunked_gelf` | | Option 12 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 12 | mode [*required*] | enum | Protocol used to receive logs. Allowed enum values: `tcp,udp` | | Option 12 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 12 | type [*required*] | enum | The source type. The value should always be `socket`. Allowed enum values: `socket` | | sources | Option 13 | object | The `splunk_hec` source implements the Splunk HTTP Event Collector (HEC) API. | **Supported pipeline types:** logs | | Option 13 | address_key | string | Name of the environment variable or secret that holds the listen address for the HEC API. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | type [*required*] | enum | The source type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | sources | Option 14 | object | The `splunk_tcp` source receives logs from a Splunk Universal Forwarder over TCP. TLS is supported for secure transmission. | **Supported pipeline types:** logs | | Option 14 | address_key | string | Name of the environment variable or secret that holds the listen address for the Splunk TCP receiver. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | type [*required*] | enum | The source type. Always `splunk_tcp`. Allowed enum values: `splunk_tcp` | | sources | Option 15 | object | The `sumo_logic` source receives logs from Sumo Logic collectors. | **Supported pipeline types:** logs | | Option 15 | address_key | string | Name of the environment variable or secret that holds the listen address for the Sumo Logic receiver. | | Option 15 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 15 | type [*required*] | enum | The source type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | sources | Option 16 | object | The `syslog_ng` source listens for logs over TCP or UDP from a `syslog-ng` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 16 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog-ng receiver. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 16 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 16 | type [*required*] | enum | The source type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | sources | Option 17 | object | The `opentelemetry` source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP. | **Supported pipeline types:** logs, metrics | | Option 17 | grpc_address_key | string | Environment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | http_address_key | string | Environment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 17 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 17 | type [*required*] | enum | The source type. The value should always be `opentelemetry`. Allowed enum values: `opentelemetry` | | config | use_legacy_search_syntax | boolean | Set to `true` to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to `false`. The legacy syntax is deprecated and will eventually be removed. Requires Observability Pipelines Worker 2.11 or later. See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax/) for more information. | | attributes | name [*required*] | string | Name of the pipeline. | | data | id [*required*] | string | Unique identifier for the pipeline. | | data | type [*required*] | string | The resource type identifier. For pipeline resources, this should always be set to `pipelines`. | | | meta | object | Metadata about the response. | | meta | totalCount | int64 | The total number of pipelines. | {% /tab %} {% tab title="Example" %} ```json { "data": [ { "attributes": { "config": { "destinations": [ { "auth_strategy": "basic", "compression": { "algorithm": "gzip" }, "custom_key": "HTTP_AUTH_CUSTOM_HEADER", "encoding": "json", "id": "http-client-destination", "inputs": [ "filter-processor" ], "password_key": "HTTP_AUTH_PASSWORD", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "token_key": "HTTP_AUTH_TOKEN", "type": "http_client", "uri_key": "HTTP_DESTINATION_URI", "username_key": "HTTP_AUTH_USERNAME" } ], "pipeline_type": "logs", "processor_groups": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "processors": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "sources": [ { "address_key": "DATADOG_AGENT_ADDRESS", "id": "datadog-agent-source", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "type": "datadog_agent" } ], "use_legacy_search_syntax": false }, "name": "Main Observability Pipeline" }, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines" } ], "meta": { "totalCount": 42 } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Not Authorized {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" ##### ```python """ List pipelines returns "OK" response """ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) response = api_instance.list_pipelines() print(response) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=python) and then save the example to `example.py` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" python3 "example.py" ##### ```ruby # List pipelines returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new p api_instance.list_pipelines() ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=ruby) and then save the example to `example.rb` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" rb "example.rb" ##### ```go // List pipelines returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) resp, r, err := api.ListPipelines(ctx, *datadogV2.NewListPipelinesOptionalParameters()) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.ListPipelines`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.ListPipelines`:\n%s\n", responseContent) } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=go) and then save the example to `main.go` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" go run "main.go" ##### ```java // List pipelines returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; import com.datadog.api.client.v2.model.ListPipelinesResponse; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); try { ListPipelinesResponse result = apiInstance.listPipelines(); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#listPipelines"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=java) and then save the example to `Example.java` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" java "Example.java" ##### ```rust // List pipelines returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ListPipelinesOptionalParams; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; #[tokio::main] async fn main() { let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api .list_pipelines(ListPipelinesOptionalParams::default()) .await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=rust) and then save the example to `src/main.rs` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" cargo run ##### ```typescript /** * List pipelines returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); apiInstance .listPipelines() .then((data: v2.ListPipelinesResponse) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=typescript) and then save the example to `example.ts` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" tsc "example.ts" {% /tab %} ## Create a new pipeline{% #create-a-new-pipeline %} {% tab title="v2" %} | Datadog site | API endpoint | | ----------------- | ----------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/obs-pipelines/pipelines | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/obs-pipelines/pipelines | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/obs-pipelines/pipelines | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/obs-pipelines/pipelines | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/obs-pipelines/pipelines | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/obs-pipelines/pipelines | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines | ### Overview Create a new pipeline. This endpoint requires the `observability_pipelines_deploy` permission. ### Request #### Body Data (required) {% tab title="Model" %} | Parent field | Field | Type | Description | | -------------------- | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Contains the the pipeline configuration. | | data | attributes [*required*] | object | Defines the pipeline's name and its components (sources, processors, and destinations). | | attributes | config [*required*] | object | Specifies the pipeline's configuration, including its sources, processors, and destinations. | | config | destinations [*required*] | [ ] | A list of destination components where processed logs are sent. | | destinations | Option 1 | object | The `http_client` destination sends data to an HTTP endpoint. | **Supported pipeline types:** logs, metrics | | Option 1 | auth_strategy | enum | HTTP authentication strategy. Allowed enum values: `none,basic,bearer` | | Option 1 | compression | object | Compression configuration for HTTP requests. | | compression | algorithm [*required*] | enum | Compression algorithm. Allowed enum values: `gzip` | | Option 1 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 1 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json` | | Option 1 | id [*required*] | string | The unique identifier for this component. | | Option 1 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 1 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 1 | type [*required*] | enum | The destination type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 1 | uri_key | string | Name of the environment variable or secret that holds the HTTP endpoint URI. | | Option 1 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | destinations | Option 2 | object | The `amazon_opensearch` destination writes logs to Amazon OpenSearch. | **Supported pipeline types:** logs | | Option 2 | auth [*required*] | object | Authentication settings for the Amazon OpenSearch destination. The `strategy` field determines whether basic or AWS-based authentication is used. | | auth | assume_role | string | The ARN of the role to assume (used with `aws` strategy). | | auth | aws_region | string | AWS region | | auth | external_id | string | External ID for the assumed role (used with `aws` strategy). | | auth | session_name | string | Session name for the assumed role (used with `aws` strategy). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | Option 2 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 2 | bulk_index | string | The index to write logs to. | | Option 2 | id [*required*] | string | The unique identifier for this component. | | Option 2 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 2 | type [*required*] | enum | The destination type. The value should always be `amazon_opensearch`. Allowed enum values: `amazon_opensearch` | | destinations | Option 3 | object | The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | bucket [*required*] | string | S3 bucket name. | | Option 3 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 3 | id [*required*] | string | Unique identifier for the destination component. | | Option 3 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 3 | key_prefix | string | Optional prefix for object keys. | | Option 3 | region [*required*] | string | AWS region of the S3 bucket. | | Option 3 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The destination type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | destinations | Option 4 | object | The `amazon_s3_generic` destination sends your logs to an Amazon S3 bucket. | **Supported pipeline types:** logs | | Option 4 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 4 | batch_settings | object | Event batching settings | | batch_settings | batch_size | int64 | Maximum batch size in bytes. | | batch_settings | timeout_secs | int64 | Maximum number of seconds to wait before flushing the batch. | | Option 4 | bucket [*required*] | string | S3 bucket name. | | Option 4 | compression [*required*] | | Compression algorithm applied to encoded logs. | | compression | Option 1 | object | Zstd compression. | | Option 1 | algorithm [*required*] | enum | The compression type. Always `zstd`. Allowed enum values: `zstd` | | Option 1 | level [*required*] | int64 | Zstd compression level. | | compression | Option 2 | object | Gzip compression. | | Option 2 | algorithm [*required*] | enum | The compression type. Always `gzip`. Allowed enum values: `gzip` | | Option 2 | level [*required*] | int64 | Gzip compression level. | | compression | Option 3 | object | Snappy compression. | | Option 3 | algorithm [*required*] | enum | The compression type. Always `snappy`. Allowed enum values: `snappy` | | Option 4 | encoding [*required*] | | Encoding format for the destination. | | encoding | Option 1 | object | JSON encoding. | | Option 1 | type [*required*] | enum | The encoding type. Always `json`. Allowed enum values: `json` | | encoding | Option 2 | object | Parquet encoding. | | Option 2 | type [*required*] | enum | The encoding type. Always `parquet`. Allowed enum values: `parquet` | | Option 4 | id [*required*] | string | Unique identifier for the destination component. | | Option 4 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 4 | key_prefix | string | Optional prefix for object keys. | | Option 4 | region [*required*] | string | AWS region of the S3 bucket. | | Option 4 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 4 | type [*required*] | enum | The destination type. Always `amazon_s3_generic`. Allowed enum values: `amazon_s3_generic` | | destinations | Option 5 | object | The `amazon_security_lake` destination sends your logs to Amazon Security Lake. | **Supported pipeline types:** logs | | Option 5 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 5 | bucket [*required*] | string | Name of the Amazon S3 bucket in Security Lake (3-63 characters). | | Option 5 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 5 | custom_source_name [*required*] | string | Custom source name for the logs in Security Lake. | | Option 5 | id [*required*] | string | Unique identifier for the destination component. | | Option 5 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 5 | region [*required*] | string | AWS region of the S3 bucket. | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The destination type. Always `amazon_security_lake`. Allowed enum values: `amazon_security_lake` | | destinations | Option 6 | object | The `azure_storage` destination forwards logs to an Azure Blob Storage container. | **Supported pipeline types:** logs | | Option 6 | blob_prefix | string | Optional prefix for blobs written to the container. | | Option 6 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 6 | connection_string_key | string | Name of the environment variable or secret that holds the Azure Storage connection string. | | Option 6 | container_name [*required*] | string | The name of the Azure Blob Storage container to store logs in. | | Option 6 | id [*required*] | string | The unique identifier for this component. | | Option 6 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 6 | type [*required*] | enum | The destination type. The value should always be `azure_storage`. Allowed enum values: `azure_storage` | | destinations | Option 7 | object | The `cloud_prem` destination sends logs to Datadog CloudPrem. | **Supported pipeline types:** logs | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the CloudPrem endpoint URL. | | Option 7 | id [*required*] | string | The unique identifier for this component. | | Option 7 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 7 | type [*required*] | enum | The destination type. The value should always be `cloud_prem`. Allowed enum values: `cloud_prem` | | destinations | Option 8 | object | The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM. | **Supported pipeline types:** logs | | Option 8 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 8 | compression | object | Compression configuration for log events. | | compression | algorithm [*required*] | enum | Compression algorithm for log events. Allowed enum values: `gzip,zlib` | | compression | level | int64 | Compression level. | | Option 8 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 8 | endpoint_url_key | string | Name of the environment variable or secret that holds the CrowdStrike endpoint URL. | | Option 8 | id [*required*] | string | The unique identifier for this component. | | Option 8 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | token_key | string | Name of the environment variable or secret that holds the CrowdStrike API token. | | Option 8 | type [*required*] | enum | The destination type. The value should always be `crowdstrike_next_gen_siem`. Allowed enum values: `crowdstrike_next_gen_siem` | | destinations | Option 9 | object | The `datadog_logs` destination forwards logs to Datadog Log Management. | **Supported pipeline types:** logs | | Option 9 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 9 | id [*required*] | string | The unique identifier for this component. | | Option 9 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 9 | routes | [object] | A list of routing rules that forward matching logs to Datadog using dedicated API keys. | | routes | api_key_key | string | Name of the environment variable or secret that stores the Datadog API key used by this route. | | routes | include | string | A Datadog search query that determines which logs are forwarded using this route. | | routes | route_id | string | Unique identifier for this route within the destination. | | routes | site | string | Datadog site where matching logs are sent (for example, `us1`). | | Option 9 | type [*required*] | enum | The destination type. The value should always be `datadog_logs`. Allowed enum values: `datadog_logs` | | destinations | Option 10 | object | The `elasticsearch` destination writes logs to an Elasticsearch cluster. | **Supported pipeline types:** logs | | Option 10 | api_version | enum | The Elasticsearch API version to use. Set to `auto` to auto-detect. Allowed enum values: `auto,v6,v7,v8` | | Option 10 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 10 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 10 | bulk_index | string | The index to write logs to in Elasticsearch. | | Option 10 | data_stream | object | Configuration options for writing to Elasticsearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 10 | endpoint_url_key | string | Name of the environment variable or secret that holds the Elasticsearch endpoint URL. | | Option 10 | id [*required*] | string | The unique identifier for this component. | | Option 10 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 10 | type [*required*] | enum | The destination type. The value should always be `elasticsearch`. Allowed enum values: `elasticsearch` | | destinations | Option 11 | object | The `google_chronicle` destination sends logs to Google Chronicle. | **Supported pipeline types:** logs | | Option 11 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 11 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 11 | customer_id [*required*] | string | The Google Chronicle customer ID. | | Option 11 | encoding | enum | The encoding format for the logs sent to Chronicle. Allowed enum values: `json,raw_message` | | Option 11 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Chronicle endpoint URL. | | Option 11 | id [*required*] | string | The unique identifier for this component. | | Option 11 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 11 | log_type | string | The log type metadata associated with the Chronicle destination. | | Option 11 | type [*required*] | enum | The destination type. The value should always be `google_chronicle`. Allowed enum values: `google_chronicle` | | destinations | Option 12 | object | The `google_cloud_storage` destination stores logs in a Google Cloud Storage (GCS) bucket. It requires a bucket name, Google Cloud authentication, and metadata fields. | **Supported pipeline types:** logs | | Option 12 | acl | enum | Access control list setting for objects written to the bucket. Allowed enum values: `private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control` | | Option 12 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 12 | bucket [*required*] | string | Name of the GCS bucket. | | Option 12 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 12 | id [*required*] | string | Unique identifier for the destination component. | | Option 12 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 12 | key_prefix | string | Optional prefix for object keys within the GCS bucket. | | Option 12 | metadata | [object] | Custom metadata to attach to each object uploaded to the GCS bucket. | | metadata | name [*required*] | string | The metadata key. | | metadata | value [*required*] | string | The metadata value. | | Option 12 | storage_class [*required*] | enum | Storage class used for objects stored in GCS. Allowed enum values: `STANDARD,NEARLINE,COLDLINE,ARCHIVE` | | Option 12 | type [*required*] | enum | The destination type. Always `google_cloud_storage`. Allowed enum values: `google_cloud_storage` | | destinations | Option 13 | object | The `google_pubsub` destination publishes logs to a Google Cloud Pub/Sub topic. | **Supported pipeline types:** logs | | Option 13 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 13 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 13 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 13 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL. | | Option 13 | id [*required*] | string | The unique identifier for this component. | | Option 13 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 13 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub topic. | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | topic [*required*] | string | The Pub/Sub topic name to publish logs to. | | Option 13 | type [*required*] | enum | The destination type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | destinations | Option 14 | object | The `kafka` destination sends logs to Apache Kafka topics. | **Supported pipeline types:** logs | | Option 14 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 14 | compression | enum | Compression codec for Kafka messages. Allowed enum values: `none,gzip,snappy,lz4,zstd` | | Option 14 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 14 | headers_key | string | The field name to use for Kafka message headers. | | Option 14 | id [*required*] | string | The unique identifier for this component. | | Option 14 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 14 | key_field | string | The field name to use as the Kafka message key. | | Option 14 | librdkafka_options | [object] | Optional list of advanced Kafka producer configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 14 | message_timeout_ms | int64 | Maximum time in milliseconds to wait for message delivery confirmation. | | Option 14 | rate_limit_duration_secs | int64 | Duration in seconds for the rate limit window. | | Option 14 | rate_limit_num | int64 | Maximum number of messages allowed per rate limit duration. | | Option 14 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 14 | socket_timeout_ms | int64 | Socket timeout in milliseconds for network requests. | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | topic [*required*] | string | The Kafka topic name to publish logs to. | | Option 14 | type [*required*] | enum | The destination type. The value should always be `kafka`. Allowed enum values: `kafka` | | destinations | Option 15 | object | The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel. | **Supported pipeline types:** logs | | Option 15 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 15 | client_id [*required*] | string | Azure AD client ID used for authentication. | | Option 15 | client_secret_key | string | Name of the environment variable or secret that holds the Azure AD client secret. | | Option 15 | dce_uri_key | string | Name of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI. | | Option 15 | dcr_immutable_id [*required*] | string | The immutable ID of the Data Collection Rule (DCR). | | Option 15 | id [*required*] | string | The unique identifier for this component. | | Option 15 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 15 | table [*required*] | string | The name of the Log Analytics table where logs are sent. | | Option 15 | tenant_id [*required*] | string | Azure AD tenant ID. | | Option 15 | type [*required*] | enum | The destination type. The value should always be `microsoft_sentinel`. Allowed enum values: `microsoft_sentinel` | | destinations | Option 16 | object | The `new_relic` destination sends logs to the New Relic platform. | **Supported pipeline types:** logs | | Option 16 | account_id_key | string | Name of the environment variable or secret that holds the New Relic account ID. | | Option 16 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 16 | id [*required*] | string | The unique identifier for this component. | | Option 16 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 16 | license_key_key | string | Name of the environment variable or secret that holds the New Relic license key. | | Option 16 | region [*required*] | enum | The New Relic region. Allowed enum values: `us,eu` | | Option 16 | type [*required*] | enum | The destination type. The value should always be `new_relic`. Allowed enum values: `new_relic` | | destinations | Option 17 | object | The `opensearch` destination writes logs to an OpenSearch cluster. | **Supported pipeline types:** logs | | Option 17 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 17 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 17 | bulk_index | string | The index to write logs to. | | Option 17 | data_stream | object | Configuration options for writing to OpenSearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 17 | endpoint_url_key | string | Name of the environment variable or secret that holds the OpenSearch endpoint URL. | | Option 17 | id [*required*] | string | The unique identifier for this component. | | Option 17 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 17 | type [*required*] | enum | The destination type. The value should always be `opensearch`. Allowed enum values: `opensearch` | | destinations | Option 18 | object | The `rsyslog` destination forwards logs to an external `rsyslog` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 18 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 18 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog server endpoint URL. | | Option 18 | id [*required*] | string | The unique identifier for this component. | | Option 18 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 18 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 18 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 18 | type [*required*] | enum | The destination type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | destinations | Option 19 | object | The `sentinel_one` destination sends logs to SentinelOne. | **Supported pipeline types:** logs | | Option 19 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 19 | id [*required*] | string | The unique identifier for this component. | | Option 19 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 19 | region [*required*] | enum | The SentinelOne region to send logs to. Allowed enum values: `us,eu,ca,data_set_us` | | Option 19 | token_key | string | Name of the environment variable or secret that holds the SentinelOne API token. | | Option 19 | type [*required*] | enum | The destination type. The value should always be `sentinel_one`. Allowed enum values: `sentinel_one` | | destinations | Option 20 | object | The `socket` destination sends logs over TCP or UDP to a remote server. | **Supported pipeline types:** logs | | Option 20 | address_key | string | Name of the environment variable or secret that holds the socket address (host:port). | | Option 20 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 20 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 20 | framing [*required*] | | Framing method configuration. | | framing | Option 1 | object | Each log event is delimited by a newline character. | | Option 1 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod` object. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Event data is not delimited at all. | | Option 2 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod` object. Allowed enum values: `bytes` | | framing | Option 3 | object | Each log event is separated using the specified delimiter character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used as a delimiter. | | Option 3 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod` object. Allowed enum values: `character_delimited` | | Option 20 | id [*required*] | string | The unique identifier for this component. | | Option 20 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 20 | mode [*required*] | enum | Protocol used to send logs. Allowed enum values: `tcp,udp` | | Option 20 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 20 | type [*required*] | enum | The destination type. The value should always be `socket`. Allowed enum values: `socket` | | destinations | Option 21 | object | The `splunk_hec` destination forwards logs to Splunk using the HTTP Event Collector (HEC). | **Supported pipeline types:** logs | | Option 21 | auto_extract_timestamp | boolean | If `true`, Splunk tries to extract timestamps from incoming log events. If `false`, Splunk assigns the time the event was received. | | Option 21 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 21 | encoding | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 21 | endpoint_url_key | string | Name of the environment variable or secret that holds the Splunk HEC endpoint URL. | | Option 21 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 21 | index | string | Optional name of the Splunk index where logs are written. | | Option 21 | indexed_fields | [string] | List of log field names to send as indexed fields to Splunk HEC. Available only when `encoding` is `json`. | | Option 21 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 21 | sourcetype | string | The Splunk sourcetype to assign to log events. | | Option 21 | token_key | string | Name of the environment variable or secret that holds the Splunk HEC token. | | Option 21 | type [*required*] | enum | The destination type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | destinations | Option 22 | object | The `sumo_logic` destination forwards logs to Sumo Logic. | **Supported pipeline types:** logs | | Option 22 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 22 | encoding | enum | The output encoding format. Allowed enum values: `json,raw_message,logfmt` | | Option 22 | endpoint_url_key | string | Name of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL. | | Option 22 | header_custom_fields | [object] | A list of custom headers to include in the request to Sumo Logic. | | header_custom_fields | name [*required*] | string | The header field name. | | header_custom_fields | value [*required*] | string | The header field value. | | Option 22 | header_host_name | string | Optional override for the host name header. | | Option 22 | header_source_category | string | Optional override for the source category header. | | Option 22 | header_source_name | string | Optional override for the source name header. | | Option 22 | id [*required*] | string | The unique identifier for this component. | | Option 22 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 22 | type [*required*] | enum | The destination type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | destinations | Option 23 | object | The `syslog_ng` destination forwards logs to an external `syslog-ng` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 23 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 23 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog-ng server endpoint URL. | | Option 23 | id [*required*] | string | The unique identifier for this component. | | Option 23 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 23 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 23 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 23 | type [*required*] | enum | The destination type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | destinations | Option 24 | object | The `datadog_metrics` destination forwards metrics to Datadog. | **Supported pipeline types:** metrics | | Option 24 | id [*required*] | string | The unique identifier for this component. | | Option 24 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 24 | type [*required*] | enum | The destination type. The value should always be `datadog_metrics`. Allowed enum values: `datadog_metrics` | | config | pipeline_type | enum | The type of data being ingested. Defaults to `logs` if not specified. Allowed enum values: `logs,metrics` | | config | processor_groups | [object] | A list of processor groups that transform or enrich log data. | | processor_groups | display_name | string | The display name for a component. | | processor_groups | enabled [*required*] | boolean | Whether this processor group is enabled. | | processor_groups | id [*required*] | string | The unique identifier for the processor group. | | processor_groups | include [*required*] | string | Conditional expression for when this processor group should execute. | | processor_groups | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processor_groups | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | processors | [object] | **DEPRECATED**: A list of processor groups that transform or enrich log data. | **Deprecated:** This field is deprecated, you should now use the processor_groups field. | | processors | display_name | string | The display name for a component. | | processors | enabled [*required*] | boolean | Whether this processor group is enabled. | | processors | id [*required*] | string | The unique identifier for the processor group. | | processors | include [*required*] | string | Conditional expression for when this processor group should execute. | | processors | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processors | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | sources [*required*] | [ ] | A list of configured data sources for the pipeline. | | sources | Option 1 | object | The `datadog_agent` source collects logs/metrics from the Datadog Agent. | **Supported pipeline types:** logs, metrics | | Option 1 | address_key | string | Name of the environment variable or secret that holds the listen address for the Datadog Agent source. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | type [*required*] | enum | The source type. The value should always be `datadog_agent`. Allowed enum values: `datadog_agent` | | sources | Option 2 | object | The `amazon_data_firehose` source ingests logs from AWS Data Firehose. | **Supported pipeline types:** logs | | Option 2 | address_key | string | Name of the environment variable or secret that holds the Firehose delivery stream address. | | Option 2 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 2 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 2 | type [*required*] | enum | The source type. The value should always be `amazon_data_firehose`. Allowed enum values: `amazon_data_firehose` | | sources | Option 3 | object | The `amazon_s3` source ingests logs from an Amazon S3 bucket. It supports AWS authentication and TLS encryption. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | region [*required*] | string | AWS region where the S3 bucket resides. | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The source type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | Option 3 | url_key | string | Name of the environment variable or secret that holds the S3 bucket URL. | | sources | Option 4 | object | The `fluent_bit` source ingests logs from Fluent Bit. | **Supported pipeline types:** logs | | Option 4 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent Bit receiver. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 4 | type [*required*] | enum | The source type. The value should always be `fluent_bit`. Allowed enum values: `fluent_bit` | | sources | Option 5 | object | The `fluentd` source ingests logs from a Fluentd-compatible service. | **Supported pipeline types:** logs | | Option 5 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent receiver. | | Option 5 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The source type. The value should always be `fluentd. Allowed enum values: `fluentd` | | sources | Option 6 | object | The `google_pubsub` source ingests logs from a Google Cloud Pub/Sub subscription. | **Supported pipeline types:** logs | | Option 6 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 6 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub subscription. | | Option 6 | subscription [*required*] | string | The Pub/Sub subscription name from which messages are consumed. | | Option 6 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 6 | type [*required*] | enum | The source type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | sources | Option 7 | object | The `http_client` source scrapes logs from HTTP endpoints at regular intervals. | **Supported pipeline types:** logs | | Option 7 | auth_strategy | enum | Optional authentication strategy for HTTP requests. Allowed enum values: `none,basic,bearer,custom` | | Option 7 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 7 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the HTTP endpoint URL to scrape. | | Option 7 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 7 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 7 | scrape_interval_secs | int64 | The interval (in seconds) between HTTP scrape requests. | | Option 7 | scrape_timeout_secs | int64 | The timeout (in seconds) for each scrape request. | | Option 7 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 7 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 7 | type [*required*] | enum | The source type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 7 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | sources | Option 8 | object | The `http_server` source collects logs over HTTP POST from external services. | **Supported pipeline types:** logs | | Option 8 | address_key | string | Name of the environment variable or secret that holds the listen address for the HTTP server. | | Option 8 | auth_strategy [*required*] | enum | HTTP authentication method. Allowed enum values: `none,plain` | | Option 8 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 8 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 8 | id [*required*] | string | Unique ID for the HTTP server source. | | Option 8 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `plain`). | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | type [*required*] | enum | The source type. The value should always be `http_server`. Allowed enum values: `http_server` | | Option 8 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `plain`). | | sources | Option 9 | object | The `kafka` source ingests data from Apache Kafka topics. | **Supported pipeline types:** logs | | Option 9 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 9 | group_id [*required*] | string | Consumer group ID used by the Kafka client. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 9 | librdkafka_options | [object] | Optional list of advanced Kafka client configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 9 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 9 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 9 | topics [*required*] | [string] | A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified. | | Option 9 | type [*required*] | enum | The source type. The value should always be `kafka`. Allowed enum values: `kafka` | | sources | Option 10 | object | The `logstash` source ingests logs from a Logstash forwarder. | **Supported pipeline types:** logs | | Option 10 | address_key | string | Name of the environment variable or secret that holds the listen address for the Logstash receiver. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 10 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 10 | type [*required*] | enum | The source type. The value should always be `logstash`. Allowed enum values: `logstash` | | sources | Option 11 | object | The `rsyslog` source listens for logs over TCP or UDP from an `rsyslog` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 11 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog receiver. | | Option 11 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 11 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 11 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 11 | type [*required*] | enum | The source type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | sources | Option 12 | object | The `socket` source ingests logs over TCP or UDP. | **Supported pipeline types:** logs | | Option 12 | address_key | string | Name of the environment variable or secret that holds the listen address for the socket. | | Option 12 | framing [*required*] | | Framing method configuration for the socket source. | | framing | Option 1 | object | Byte frames which are delimited by a newline character. | | Option 1 | method [*required*] | enum | Byte frames which are delimited by a newline character. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). | | Option 2 | method [*required*] | enum | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). Allowed enum values: `bytes` | | framing | Option 3 | object | Byte frames which are delimited by a chosen character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used to delimit events. | | Option 3 | method [*required*] | enum | Byte frames which are delimited by a chosen character. Allowed enum values: `character_delimited` | | framing | Option 4 | object | Byte frames according to the octet counting format as per RFC6587. | | Option 4 | method [*required*] | enum | Byte frames according to the octet counting format as per RFC6587. Allowed enum values: `octet_counting` | | framing | Option 5 | object | Byte frames which are chunked GELF messages. | | Option 5 | method [*required*] | enum | Byte frames which are chunked GELF messages. Allowed enum values: `chunked_gelf` | | Option 12 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 12 | mode [*required*] | enum | Protocol used to receive logs. Allowed enum values: `tcp,udp` | | Option 12 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 12 | type [*required*] | enum | The source type. The value should always be `socket`. Allowed enum values: `socket` | | sources | Option 13 | object | The `splunk_hec` source implements the Splunk HTTP Event Collector (HEC) API. | **Supported pipeline types:** logs | | Option 13 | address_key | string | Name of the environment variable or secret that holds the listen address for the HEC API. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | type [*required*] | enum | The source type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | sources | Option 14 | object | The `splunk_tcp` source receives logs from a Splunk Universal Forwarder over TCP. TLS is supported for secure transmission. | **Supported pipeline types:** logs | | Option 14 | address_key | string | Name of the environment variable or secret that holds the listen address for the Splunk TCP receiver. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | type [*required*] | enum | The source type. Always `splunk_tcp`. Allowed enum values: `splunk_tcp` | | sources | Option 15 | object | The `sumo_logic` source receives logs from Sumo Logic collectors. | **Supported pipeline types:** logs | | Option 15 | address_key | string | Name of the environment variable or secret that holds the listen address for the Sumo Logic receiver. | | Option 15 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 15 | type [*required*] | enum | The source type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | sources | Option 16 | object | The `syslog_ng` source listens for logs over TCP or UDP from a `syslog-ng` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 16 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog-ng receiver. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 16 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 16 | type [*required*] | enum | The source type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | sources | Option 17 | object | The `opentelemetry` source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP. | **Supported pipeline types:** logs, metrics | | Option 17 | grpc_address_key | string | Environment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | http_address_key | string | Environment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 17 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 17 | type [*required*] | enum | The source type. The value should always be `opentelemetry`. Allowed enum values: `opentelemetry` | | config | use_legacy_search_syntax | boolean | Set to `true` to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to `false`. The legacy syntax is deprecated and will eventually be removed. Requires Observability Pipelines Worker 2.11 or later. See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax/) for more information. | | attributes | name [*required*] | string | Name of the pipeline. | | data | type [*required*] | string | The resource type identifier. For pipeline resources, this should always be set to `pipelines`. | {% /tab %} {% tab title="Example" %} ##### ```json { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Main Observability Pipeline" }, "type": "pipelines" } } ``` ##### ```json { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "dedupe-processor", "include": "service:my-service", "type": "dedupe", "fields": [ "message" ], "mode": "match", "cache": { "num_events": 5000 } } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Pipeline with Dedupe Cache" }, "type": "pipelines" } } ``` ##### ```json { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "dedupe-processor", "include": "service:my-service", "type": "dedupe", "fields": [ "message" ], "mode": "match" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Pipeline with Dedupe No Cache" }, "type": "pipelines" } } ``` {% /tab %} ### Response {% tab title="201" %} OK {% tab title="Model" %} Top-level schema representing a pipeline. | Parent field | Field | Type | Description | | -------------------- | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Contains the pipeline's ID, type, and configuration attributes. | | data | attributes [*required*] | object | Defines the pipeline's name and its components (sources, processors, and destinations). | | attributes | config [*required*] | object | Specifies the pipeline's configuration, including its sources, processors, and destinations. | | config | destinations [*required*] | [ ] | A list of destination components where processed logs are sent. | | destinations | Option 1 | object | The `http_client` destination sends data to an HTTP endpoint. | **Supported pipeline types:** logs, metrics | | Option 1 | auth_strategy | enum | HTTP authentication strategy. Allowed enum values: `none,basic,bearer` | | Option 1 | compression | object | Compression configuration for HTTP requests. | | compression | algorithm [*required*] | enum | Compression algorithm. Allowed enum values: `gzip` | | Option 1 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 1 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json` | | Option 1 | id [*required*] | string | The unique identifier for this component. | | Option 1 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 1 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 1 | type [*required*] | enum | The destination type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 1 | uri_key | string | Name of the environment variable or secret that holds the HTTP endpoint URI. | | Option 1 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | destinations | Option 2 | object | The `amazon_opensearch` destination writes logs to Amazon OpenSearch. | **Supported pipeline types:** logs | | Option 2 | auth [*required*] | object | Authentication settings for the Amazon OpenSearch destination. The `strategy` field determines whether basic or AWS-based authentication is used. | | auth | assume_role | string | The ARN of the role to assume (used with `aws` strategy). | | auth | aws_region | string | AWS region | | auth | external_id | string | External ID for the assumed role (used with `aws` strategy). | | auth | session_name | string | Session name for the assumed role (used with `aws` strategy). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | Option 2 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 2 | bulk_index | string | The index to write logs to. | | Option 2 | id [*required*] | string | The unique identifier for this component. | | Option 2 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 2 | type [*required*] | enum | The destination type. The value should always be `amazon_opensearch`. Allowed enum values: `amazon_opensearch` | | destinations | Option 3 | object | The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | bucket [*required*] | string | S3 bucket name. | | Option 3 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 3 | id [*required*] | string | Unique identifier for the destination component. | | Option 3 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 3 | key_prefix | string | Optional prefix for object keys. | | Option 3 | region [*required*] | string | AWS region of the S3 bucket. | | Option 3 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The destination type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | destinations | Option 4 | object | The `amazon_s3_generic` destination sends your logs to an Amazon S3 bucket. | **Supported pipeline types:** logs | | Option 4 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 4 | batch_settings | object | Event batching settings | | batch_settings | batch_size | int64 | Maximum batch size in bytes. | | batch_settings | timeout_secs | int64 | Maximum number of seconds to wait before flushing the batch. | | Option 4 | bucket [*required*] | string | S3 bucket name. | | Option 4 | compression [*required*] | | Compression algorithm applied to encoded logs. | | compression | Option 1 | object | Zstd compression. | | Option 1 | algorithm [*required*] | enum | The compression type. Always `zstd`. Allowed enum values: `zstd` | | Option 1 | level [*required*] | int64 | Zstd compression level. | | compression | Option 2 | object | Gzip compression. | | Option 2 | algorithm [*required*] | enum | The compression type. Always `gzip`. Allowed enum values: `gzip` | | Option 2 | level [*required*] | int64 | Gzip compression level. | | compression | Option 3 | object | Snappy compression. | | Option 3 | algorithm [*required*] | enum | The compression type. Always `snappy`. Allowed enum values: `snappy` | | Option 4 | encoding [*required*] | | Encoding format for the destination. | | encoding | Option 1 | object | JSON encoding. | | Option 1 | type [*required*] | enum | The encoding type. Always `json`. Allowed enum values: `json` | | encoding | Option 2 | object | Parquet encoding. | | Option 2 | type [*required*] | enum | The encoding type. Always `parquet`. Allowed enum values: `parquet` | | Option 4 | id [*required*] | string | Unique identifier for the destination component. | | Option 4 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 4 | key_prefix | string | Optional prefix for object keys. | | Option 4 | region [*required*] | string | AWS region of the S3 bucket. | | Option 4 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 4 | type [*required*] | enum | The destination type. Always `amazon_s3_generic`. Allowed enum values: `amazon_s3_generic` | | destinations | Option 5 | object | The `amazon_security_lake` destination sends your logs to Amazon Security Lake. | **Supported pipeline types:** logs | | Option 5 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 5 | bucket [*required*] | string | Name of the Amazon S3 bucket in Security Lake (3-63 characters). | | Option 5 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 5 | custom_source_name [*required*] | string | Custom source name for the logs in Security Lake. | | Option 5 | id [*required*] | string | Unique identifier for the destination component. | | Option 5 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 5 | region [*required*] | string | AWS region of the S3 bucket. | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The destination type. Always `amazon_security_lake`. Allowed enum values: `amazon_security_lake` | | destinations | Option 6 | object | The `azure_storage` destination forwards logs to an Azure Blob Storage container. | **Supported pipeline types:** logs | | Option 6 | blob_prefix | string | Optional prefix for blobs written to the container. | | Option 6 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 6 | connection_string_key | string | Name of the environment variable or secret that holds the Azure Storage connection string. | | Option 6 | container_name [*required*] | string | The name of the Azure Blob Storage container to store logs in. | | Option 6 | id [*required*] | string | The unique identifier for this component. | | Option 6 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 6 | type [*required*] | enum | The destination type. The value should always be `azure_storage`. Allowed enum values: `azure_storage` | | destinations | Option 7 | object | The `cloud_prem` destination sends logs to Datadog CloudPrem. | **Supported pipeline types:** logs | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the CloudPrem endpoint URL. | | Option 7 | id [*required*] | string | The unique identifier for this component. | | Option 7 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 7 | type [*required*] | enum | The destination type. The value should always be `cloud_prem`. Allowed enum values: `cloud_prem` | | destinations | Option 8 | object | The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM. | **Supported pipeline types:** logs | | Option 8 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 8 | compression | object | Compression configuration for log events. | | compression | algorithm [*required*] | enum | Compression algorithm for log events. Allowed enum values: `gzip,zlib` | | compression | level | int64 | Compression level. | | Option 8 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 8 | endpoint_url_key | string | Name of the environment variable or secret that holds the CrowdStrike endpoint URL. | | Option 8 | id [*required*] | string | The unique identifier for this component. | | Option 8 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | token_key | string | Name of the environment variable or secret that holds the CrowdStrike API token. | | Option 8 | type [*required*] | enum | The destination type. The value should always be `crowdstrike_next_gen_siem`. Allowed enum values: `crowdstrike_next_gen_siem` | | destinations | Option 9 | object | The `datadog_logs` destination forwards logs to Datadog Log Management. | **Supported pipeline types:** logs | | Option 9 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 9 | id [*required*] | string | The unique identifier for this component. | | Option 9 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 9 | routes | [object] | A list of routing rules that forward matching logs to Datadog using dedicated API keys. | | routes | api_key_key | string | Name of the environment variable or secret that stores the Datadog API key used by this route. | | routes | include | string | A Datadog search query that determines which logs are forwarded using this route. | | routes | route_id | string | Unique identifier for this route within the destination. | | routes | site | string | Datadog site where matching logs are sent (for example, `us1`). | | Option 9 | type [*required*] | enum | The destination type. The value should always be `datadog_logs`. Allowed enum values: `datadog_logs` | | destinations | Option 10 | object | The `elasticsearch` destination writes logs to an Elasticsearch cluster. | **Supported pipeline types:** logs | | Option 10 | api_version | enum | The Elasticsearch API version to use. Set to `auto` to auto-detect. Allowed enum values: `auto,v6,v7,v8` | | Option 10 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 10 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 10 | bulk_index | string | The index to write logs to in Elasticsearch. | | Option 10 | data_stream | object | Configuration options for writing to Elasticsearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 10 | endpoint_url_key | string | Name of the environment variable or secret that holds the Elasticsearch endpoint URL. | | Option 10 | id [*required*] | string | The unique identifier for this component. | | Option 10 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 10 | type [*required*] | enum | The destination type. The value should always be `elasticsearch`. Allowed enum values: `elasticsearch` | | destinations | Option 11 | object | The `google_chronicle` destination sends logs to Google Chronicle. | **Supported pipeline types:** logs | | Option 11 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 11 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 11 | customer_id [*required*] | string | The Google Chronicle customer ID. | | Option 11 | encoding | enum | The encoding format for the logs sent to Chronicle. Allowed enum values: `json,raw_message` | | Option 11 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Chronicle endpoint URL. | | Option 11 | id [*required*] | string | The unique identifier for this component. | | Option 11 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 11 | log_type | string | The log type metadata associated with the Chronicle destination. | | Option 11 | type [*required*] | enum | The destination type. The value should always be `google_chronicle`. Allowed enum values: `google_chronicle` | | destinations | Option 12 | object | The `google_cloud_storage` destination stores logs in a Google Cloud Storage (GCS) bucket. It requires a bucket name, Google Cloud authentication, and metadata fields. | **Supported pipeline types:** logs | | Option 12 | acl | enum | Access control list setting for objects written to the bucket. Allowed enum values: `private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control` | | Option 12 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 12 | bucket [*required*] | string | Name of the GCS bucket. | | Option 12 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 12 | id [*required*] | string | Unique identifier for the destination component. | | Option 12 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 12 | key_prefix | string | Optional prefix for object keys within the GCS bucket. | | Option 12 | metadata | [object] | Custom metadata to attach to each object uploaded to the GCS bucket. | | metadata | name [*required*] | string | The metadata key. | | metadata | value [*required*] | string | The metadata value. | | Option 12 | storage_class [*required*] | enum | Storage class used for objects stored in GCS. Allowed enum values: `STANDARD,NEARLINE,COLDLINE,ARCHIVE` | | Option 12 | type [*required*] | enum | The destination type. Always `google_cloud_storage`. Allowed enum values: `google_cloud_storage` | | destinations | Option 13 | object | The `google_pubsub` destination publishes logs to a Google Cloud Pub/Sub topic. | **Supported pipeline types:** logs | | Option 13 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 13 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 13 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 13 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL. | | Option 13 | id [*required*] | string | The unique identifier for this component. | | Option 13 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 13 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub topic. | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | topic [*required*] | string | The Pub/Sub topic name to publish logs to. | | Option 13 | type [*required*] | enum | The destination type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | destinations | Option 14 | object | The `kafka` destination sends logs to Apache Kafka topics. | **Supported pipeline types:** logs | | Option 14 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 14 | compression | enum | Compression codec for Kafka messages. Allowed enum values: `none,gzip,snappy,lz4,zstd` | | Option 14 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 14 | headers_key | string | The field name to use for Kafka message headers. | | Option 14 | id [*required*] | string | The unique identifier for this component. | | Option 14 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 14 | key_field | string | The field name to use as the Kafka message key. | | Option 14 | librdkafka_options | [object] | Optional list of advanced Kafka producer configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 14 | message_timeout_ms | int64 | Maximum time in milliseconds to wait for message delivery confirmation. | | Option 14 | rate_limit_duration_secs | int64 | Duration in seconds for the rate limit window. | | Option 14 | rate_limit_num | int64 | Maximum number of messages allowed per rate limit duration. | | Option 14 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 14 | socket_timeout_ms | int64 | Socket timeout in milliseconds for network requests. | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | topic [*required*] | string | The Kafka topic name to publish logs to. | | Option 14 | type [*required*] | enum | The destination type. The value should always be `kafka`. Allowed enum values: `kafka` | | destinations | Option 15 | object | The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel. | **Supported pipeline types:** logs | | Option 15 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 15 | client_id [*required*] | string | Azure AD client ID used for authentication. | | Option 15 | client_secret_key | string | Name of the environment variable or secret that holds the Azure AD client secret. | | Option 15 | dce_uri_key | string | Name of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI. | | Option 15 | dcr_immutable_id [*required*] | string | The immutable ID of the Data Collection Rule (DCR). | | Option 15 | id [*required*] | string | The unique identifier for this component. | | Option 15 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 15 | table [*required*] | string | The name of the Log Analytics table where logs are sent. | | Option 15 | tenant_id [*required*] | string | Azure AD tenant ID. | | Option 15 | type [*required*] | enum | The destination type. The value should always be `microsoft_sentinel`. Allowed enum values: `microsoft_sentinel` | | destinations | Option 16 | object | The `new_relic` destination sends logs to the New Relic platform. | **Supported pipeline types:** logs | | Option 16 | account_id_key | string | Name of the environment variable or secret that holds the New Relic account ID. | | Option 16 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 16 | id [*required*] | string | The unique identifier for this component. | | Option 16 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 16 | license_key_key | string | Name of the environment variable or secret that holds the New Relic license key. | | Option 16 | region [*required*] | enum | The New Relic region. Allowed enum values: `us,eu` | | Option 16 | type [*required*] | enum | The destination type. The value should always be `new_relic`. Allowed enum values: `new_relic` | | destinations | Option 17 | object | The `opensearch` destination writes logs to an OpenSearch cluster. | **Supported pipeline types:** logs | | Option 17 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 17 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 17 | bulk_index | string | The index to write logs to. | | Option 17 | data_stream | object | Configuration options for writing to OpenSearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 17 | endpoint_url_key | string | Name of the environment variable or secret that holds the OpenSearch endpoint URL. | | Option 17 | id [*required*] | string | The unique identifier for this component. | | Option 17 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 17 | type [*required*] | enum | The destination type. The value should always be `opensearch`. Allowed enum values: `opensearch` | | destinations | Option 18 | object | The `rsyslog` destination forwards logs to an external `rsyslog` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 18 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 18 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog server endpoint URL. | | Option 18 | id [*required*] | string | The unique identifier for this component. | | Option 18 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 18 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 18 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 18 | type [*required*] | enum | The destination type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | destinations | Option 19 | object | The `sentinel_one` destination sends logs to SentinelOne. | **Supported pipeline types:** logs | | Option 19 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 19 | id [*required*] | string | The unique identifier for this component. | | Option 19 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 19 | region [*required*] | enum | The SentinelOne region to send logs to. Allowed enum values: `us,eu,ca,data_set_us` | | Option 19 | token_key | string | Name of the environment variable or secret that holds the SentinelOne API token. | | Option 19 | type [*required*] | enum | The destination type. The value should always be `sentinel_one`. Allowed enum values: `sentinel_one` | | destinations | Option 20 | object | The `socket` destination sends logs over TCP or UDP to a remote server. | **Supported pipeline types:** logs | | Option 20 | address_key | string | Name of the environment variable or secret that holds the socket address (host:port). | | Option 20 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 20 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 20 | framing [*required*] | | Framing method configuration. | | framing | Option 1 | object | Each log event is delimited by a newline character. | | Option 1 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod` object. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Event data is not delimited at all. | | Option 2 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod` object. Allowed enum values: `bytes` | | framing | Option 3 | object | Each log event is separated using the specified delimiter character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used as a delimiter. | | Option 3 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod` object. Allowed enum values: `character_delimited` | | Option 20 | id [*required*] | string | The unique identifier for this component. | | Option 20 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 20 | mode [*required*] | enum | Protocol used to send logs. Allowed enum values: `tcp,udp` | | Option 20 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 20 | type [*required*] | enum | The destination type. The value should always be `socket`. Allowed enum values: `socket` | | destinations | Option 21 | object | The `splunk_hec` destination forwards logs to Splunk using the HTTP Event Collector (HEC). | **Supported pipeline types:** logs | | Option 21 | auto_extract_timestamp | boolean | If `true`, Splunk tries to extract timestamps from incoming log events. If `false`, Splunk assigns the time the event was received. | | Option 21 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 21 | encoding | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 21 | endpoint_url_key | string | Name of the environment variable or secret that holds the Splunk HEC endpoint URL. | | Option 21 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 21 | index | string | Optional name of the Splunk index where logs are written. | | Option 21 | indexed_fields | [string] | List of log field names to send as indexed fields to Splunk HEC. Available only when `encoding` is `json`. | | Option 21 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 21 | sourcetype | string | The Splunk sourcetype to assign to log events. | | Option 21 | token_key | string | Name of the environment variable or secret that holds the Splunk HEC token. | | Option 21 | type [*required*] | enum | The destination type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | destinations | Option 22 | object | The `sumo_logic` destination forwards logs to Sumo Logic. | **Supported pipeline types:** logs | | Option 22 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 22 | encoding | enum | The output encoding format. Allowed enum values: `json,raw_message,logfmt` | | Option 22 | endpoint_url_key | string | Name of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL. | | Option 22 | header_custom_fields | [object] | A list of custom headers to include in the request to Sumo Logic. | | header_custom_fields | name [*required*] | string | The header field name. | | header_custom_fields | value [*required*] | string | The header field value. | | Option 22 | header_host_name | string | Optional override for the host name header. | | Option 22 | header_source_category | string | Optional override for the source category header. | | Option 22 | header_source_name | string | Optional override for the source name header. | | Option 22 | id [*required*] | string | The unique identifier for this component. | | Option 22 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 22 | type [*required*] | enum | The destination type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | destinations | Option 23 | object | The `syslog_ng` destination forwards logs to an external `syslog-ng` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 23 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 23 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog-ng server endpoint URL. | | Option 23 | id [*required*] | string | The unique identifier for this component. | | Option 23 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 23 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 23 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 23 | type [*required*] | enum | The destination type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | destinations | Option 24 | object | The `datadog_metrics` destination forwards metrics to Datadog. | **Supported pipeline types:** metrics | | Option 24 | id [*required*] | string | The unique identifier for this component. | | Option 24 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 24 | type [*required*] | enum | The destination type. The value should always be `datadog_metrics`. Allowed enum values: `datadog_metrics` | | config | pipeline_type | enum | The type of data being ingested. Defaults to `logs` if not specified. Allowed enum values: `logs,metrics` | | config | processor_groups | [object] | A list of processor groups that transform or enrich log data. | | processor_groups | display_name | string | The display name for a component. | | processor_groups | enabled [*required*] | boolean | Whether this processor group is enabled. | | processor_groups | id [*required*] | string | The unique identifier for the processor group. | | processor_groups | include [*required*] | string | Conditional expression for when this processor group should execute. | | processor_groups | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processor_groups | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | processors | [object] | **DEPRECATED**: A list of processor groups that transform or enrich log data. | **Deprecated:** This field is deprecated, you should now use the processor_groups field. | | processors | display_name | string | The display name for a component. | | processors | enabled [*required*] | boolean | Whether this processor group is enabled. | | processors | id [*required*] | string | The unique identifier for the processor group. | | processors | include [*required*] | string | Conditional expression for when this processor group should execute. | | processors | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processors | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | sources [*required*] | [ ] | A list of configured data sources for the pipeline. | | sources | Option 1 | object | The `datadog_agent` source collects logs/metrics from the Datadog Agent. | **Supported pipeline types:** logs, metrics | | Option 1 | address_key | string | Name of the environment variable or secret that holds the listen address for the Datadog Agent source. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | type [*required*] | enum | The source type. The value should always be `datadog_agent`. Allowed enum values: `datadog_agent` | | sources | Option 2 | object | The `amazon_data_firehose` source ingests logs from AWS Data Firehose. | **Supported pipeline types:** logs | | Option 2 | address_key | string | Name of the environment variable or secret that holds the Firehose delivery stream address. | | Option 2 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 2 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 2 | type [*required*] | enum | The source type. The value should always be `amazon_data_firehose`. Allowed enum values: `amazon_data_firehose` | | sources | Option 3 | object | The `amazon_s3` source ingests logs from an Amazon S3 bucket. It supports AWS authentication and TLS encryption. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | region [*required*] | string | AWS region where the S3 bucket resides. | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The source type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | Option 3 | url_key | string | Name of the environment variable or secret that holds the S3 bucket URL. | | sources | Option 4 | object | The `fluent_bit` source ingests logs from Fluent Bit. | **Supported pipeline types:** logs | | Option 4 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent Bit receiver. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 4 | type [*required*] | enum | The source type. The value should always be `fluent_bit`. Allowed enum values: `fluent_bit` | | sources | Option 5 | object | The `fluentd` source ingests logs from a Fluentd-compatible service. | **Supported pipeline types:** logs | | Option 5 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent receiver. | | Option 5 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The source type. The value should always be `fluentd. Allowed enum values: `fluentd` | | sources | Option 6 | object | The `google_pubsub` source ingests logs from a Google Cloud Pub/Sub subscription. | **Supported pipeline types:** logs | | Option 6 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 6 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub subscription. | | Option 6 | subscription [*required*] | string | The Pub/Sub subscription name from which messages are consumed. | | Option 6 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 6 | type [*required*] | enum | The source type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | sources | Option 7 | object | The `http_client` source scrapes logs from HTTP endpoints at regular intervals. | **Supported pipeline types:** logs | | Option 7 | auth_strategy | enum | Optional authentication strategy for HTTP requests. Allowed enum values: `none,basic,bearer,custom` | | Option 7 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 7 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the HTTP endpoint URL to scrape. | | Option 7 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 7 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 7 | scrape_interval_secs | int64 | The interval (in seconds) between HTTP scrape requests. | | Option 7 | scrape_timeout_secs | int64 | The timeout (in seconds) for each scrape request. | | Option 7 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 7 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 7 | type [*required*] | enum | The source type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 7 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | sources | Option 8 | object | The `http_server` source collects logs over HTTP POST from external services. | **Supported pipeline types:** logs | | Option 8 | address_key | string | Name of the environment variable or secret that holds the listen address for the HTTP server. | | Option 8 | auth_strategy [*required*] | enum | HTTP authentication method. Allowed enum values: `none,plain` | | Option 8 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 8 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 8 | id [*required*] | string | Unique ID for the HTTP server source. | | Option 8 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `plain`). | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | type [*required*] | enum | The source type. The value should always be `http_server`. Allowed enum values: `http_server` | | Option 8 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `plain`). | | sources | Option 9 | object | The `kafka` source ingests data from Apache Kafka topics. | **Supported pipeline types:** logs | | Option 9 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 9 | group_id [*required*] | string | Consumer group ID used by the Kafka client. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 9 | librdkafka_options | [object] | Optional list of advanced Kafka client configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 9 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 9 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 9 | topics [*required*] | [string] | A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified. | | Option 9 | type [*required*] | enum | The source type. The value should always be `kafka`. Allowed enum values: `kafka` | | sources | Option 10 | object | The `logstash` source ingests logs from a Logstash forwarder. | **Supported pipeline types:** logs | | Option 10 | address_key | string | Name of the environment variable or secret that holds the listen address for the Logstash receiver. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 10 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 10 | type [*required*] | enum | The source type. The value should always be `logstash`. Allowed enum values: `logstash` | | sources | Option 11 | object | The `rsyslog` source listens for logs over TCP or UDP from an `rsyslog` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 11 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog receiver. | | Option 11 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 11 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 11 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 11 | type [*required*] | enum | The source type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | sources | Option 12 | object | The `socket` source ingests logs over TCP or UDP. | **Supported pipeline types:** logs | | Option 12 | address_key | string | Name of the environment variable or secret that holds the listen address for the socket. | | Option 12 | framing [*required*] | | Framing method configuration for the socket source. | | framing | Option 1 | object | Byte frames which are delimited by a newline character. | | Option 1 | method [*required*] | enum | Byte frames which are delimited by a newline character. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). | | Option 2 | method [*required*] | enum | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). Allowed enum values: `bytes` | | framing | Option 3 | object | Byte frames which are delimited by a chosen character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used to delimit events. | | Option 3 | method [*required*] | enum | Byte frames which are delimited by a chosen character. Allowed enum values: `character_delimited` | | framing | Option 4 | object | Byte frames according to the octet counting format as per RFC6587. | | Option 4 | method [*required*] | enum | Byte frames according to the octet counting format as per RFC6587. Allowed enum values: `octet_counting` | | framing | Option 5 | object | Byte frames which are chunked GELF messages. | | Option 5 | method [*required*] | enum | Byte frames which are chunked GELF messages. Allowed enum values: `chunked_gelf` | | Option 12 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 12 | mode [*required*] | enum | Protocol used to receive logs. Allowed enum values: `tcp,udp` | | Option 12 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 12 | type [*required*] | enum | The source type. The value should always be `socket`. Allowed enum values: `socket` | | sources | Option 13 | object | The `splunk_hec` source implements the Splunk HTTP Event Collector (HEC) API. | **Supported pipeline types:** logs | | Option 13 | address_key | string | Name of the environment variable or secret that holds the listen address for the HEC API. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | type [*required*] | enum | The source type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | sources | Option 14 | object | The `splunk_tcp` source receives logs from a Splunk Universal Forwarder over TCP. TLS is supported for secure transmission. | **Supported pipeline types:** logs | | Option 14 | address_key | string | Name of the environment variable or secret that holds the listen address for the Splunk TCP receiver. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | type [*required*] | enum | The source type. Always `splunk_tcp`. Allowed enum values: `splunk_tcp` | | sources | Option 15 | object | The `sumo_logic` source receives logs from Sumo Logic collectors. | **Supported pipeline types:** logs | | Option 15 | address_key | string | Name of the environment variable or secret that holds the listen address for the Sumo Logic receiver. | | Option 15 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 15 | type [*required*] | enum | The source type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | sources | Option 16 | object | The `syslog_ng` source listens for logs over TCP or UDP from a `syslog-ng` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 16 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog-ng receiver. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 16 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 16 | type [*required*] | enum | The source type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | sources | Option 17 | object | The `opentelemetry` source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP. | **Supported pipeline types:** logs, metrics | | Option 17 | grpc_address_key | string | Environment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | http_address_key | string | Environment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 17 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 17 | type [*required*] | enum | The source type. The value should always be `opentelemetry`. Allowed enum values: `opentelemetry` | | config | use_legacy_search_syntax | boolean | Set to `true` to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to `false`. The legacy syntax is deprecated and will eventually be removed. Requires Observability Pipelines Worker 2.11 or later. See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax/) for more information. | | attributes | name [*required*] | string | Name of the pipeline. | | data | id [*required*] | string | Unique identifier for the pipeline. | | data | type [*required*] | string | The resource type identifier. For pipeline resources, this should always be set to `pipelines`. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": { "destinations": [ { "auth_strategy": "basic", "compression": { "algorithm": "gzip" }, "custom_key": "HTTP_AUTH_CUSTOM_HEADER", "encoding": "json", "id": "http-client-destination", "inputs": [ "filter-processor" ], "password_key": "HTTP_AUTH_PASSWORD", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "token_key": "HTTP_AUTH_TOKEN", "type": "http_client", "uri_key": "HTTP_DESTINATION_URI", "username_key": "HTTP_AUTH_USERNAME" } ], "pipeline_type": "logs", "processor_groups": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "processors": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "sources": [ { "address_key": "DATADOG_AGENT_ADDRESS", "id": "datadog-agent-source", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "type": "datadog_agent" } ], "use_legacy_search_syntax": false }, "name": "Main Observability Pipeline" }, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Not Authorized {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="409" %} Conflict {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Main Observability Pipeline" }, "type": "pipelines" } } EOF ##### \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "dedupe-processor", "include": "service:my-service", "type": "dedupe", "fields": [ "message" ], "mode": "match", "cache": { "num_events": 5000 } } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Pipeline with Dedupe Cache" }, "type": "pipelines" } } EOF ##### \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "dedupe-processor", "include": "service:my-service", "type": "dedupe", "fields": [ "message" ], "mode": "match" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Pipeline with Dedupe No Cache" }, "type": "pipelines" } } EOF ##### ```go // Create a new pipeline returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { body := datadogV2.ObservabilityPipelineSpec{ Data: datadogV2.ObservabilityPipelineSpecData{ Attributes: datadogV2.ObservabilityPipelineDataAttributes{ Config: datadogV2.ObservabilityPipelineConfig{ Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{ datadogV2.ObservabilityPipelineConfigDestinationItem{ ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{ Id: "datadog-logs-destination", Inputs: []string{ "my-processor-group", }, Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS, }}, }, ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{ { Enabled: true, Id: "my-processor-group", Include: "service:my-service", Inputs: []string{ "datadog-agent-source", }, Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{ datadogV2.ObservabilityPipelineConfigProcessorItem{ ObservabilityPipelineFilterProcessor: &datadogV2.ObservabilityPipelineFilterProcessor{ Enabled: true, Id: "filter-processor", Include: "status:error", Type: datadogV2.OBSERVABILITYPIPELINEFILTERPROCESSORTYPE_FILTER, }}, }, }, }, Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{ datadogV2.ObservabilityPipelineConfigSourceItem{ ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{ Id: "datadog-agent-source", Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT, }}, }, }, Name: "Main Observability Pipeline", }, Type: "pipelines", }, } ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) resp, r, err := api.CreatePipeline(ctx, body) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.CreatePipeline`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.CreatePipeline`:\n%s\n", responseContent) } ``` ##### ```go // Create a pipeline with dedupe processor with cache returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { body := datadogV2.ObservabilityPipelineSpec{ Data: datadogV2.ObservabilityPipelineSpecData{ Attributes: datadogV2.ObservabilityPipelineDataAttributes{ Config: datadogV2.ObservabilityPipelineConfig{ Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{ datadogV2.ObservabilityPipelineConfigDestinationItem{ ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{ Id: "datadog-logs-destination", Inputs: []string{ "my-processor-group", }, Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS, }}, }, ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{ { Enabled: true, Id: "my-processor-group", Include: "service:my-service", Inputs: []string{ "datadog-agent-source", }, Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{ datadogV2.ObservabilityPipelineConfigProcessorItem{ ObservabilityPipelineDedupeProcessor: &datadogV2.ObservabilityPipelineDedupeProcessor{ Enabled: true, Id: "dedupe-processor", Include: "service:my-service", Type: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORTYPE_DEDUPE, Fields: []string{ "message", }, Mode: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORMODE_MATCH, Cache: &datadogV2.ObservabilityPipelineDedupeProcessorCache{ NumEvents: 5000, }, }}, }, }, }, Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{ datadogV2.ObservabilityPipelineConfigSourceItem{ ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{ Id: "datadog-agent-source", Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT, }}, }, }, Name: "Pipeline with Dedupe Cache", }, Type: "pipelines", }, } ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) resp, r, err := api.CreatePipeline(ctx, body) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.CreatePipeline`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.CreatePipeline`:\n%s\n", responseContent) } ``` ##### ```go // Create a pipeline with dedupe processor without cache returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { body := datadogV2.ObservabilityPipelineSpec{ Data: datadogV2.ObservabilityPipelineSpecData{ Attributes: datadogV2.ObservabilityPipelineDataAttributes{ Config: datadogV2.ObservabilityPipelineConfig{ Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{ datadogV2.ObservabilityPipelineConfigDestinationItem{ ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{ Id: "datadog-logs-destination", Inputs: []string{ "my-processor-group", }, Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS, }}, }, ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{ { Enabled: true, Id: "my-processor-group", Include: "service:my-service", Inputs: []string{ "datadog-agent-source", }, Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{ datadogV2.ObservabilityPipelineConfigProcessorItem{ ObservabilityPipelineDedupeProcessor: &datadogV2.ObservabilityPipelineDedupeProcessor{ Enabled: true, Id: "dedupe-processor", Include: "service:my-service", Type: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORTYPE_DEDUPE, Fields: []string{ "message", }, Mode: datadogV2.OBSERVABILITYPIPELINEDEDUPEPROCESSORMODE_MATCH, }}, }, }, }, Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{ datadogV2.ObservabilityPipelineConfigSourceItem{ ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{ Id: "datadog-agent-source", Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT, }}, }, }, Name: "Pipeline with Dedupe No Cache", }, Type: "pipelines", }, } ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) resp, r, err := api.CreatePipeline(ctx, body) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.CreatePipeline`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.CreatePipeline`:\n%s\n", responseContent) } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=go) and then save the example to `main.go` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" go run "main.go" ##### ```java // Create a new pipeline returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; import com.datadog.api.client.v2.model.ObservabilityPipeline; import com.datadog.api.client.v2.model.ObservabilityPipelineConfig; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem; import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType; import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessor; import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessorType; import com.datadog.api.client.v2.model.ObservabilityPipelineSpec; import com.datadog.api.client.v2.model.ObservabilityPipelineSpecData; import java.util.Collections; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); ObservabilityPipelineSpec body = new ObservabilityPipelineSpec() .data( new ObservabilityPipelineSpecData() .attributes( new ObservabilityPipelineDataAttributes() .config( new ObservabilityPipelineConfig() .destinations( Collections.singletonList( new ObservabilityPipelineConfigDestinationItem( new ObservabilityPipelineDatadogLogsDestination() .id("datadog-logs-destination") .inputs( Collections.singletonList( "my-processor-group")) .type( ObservabilityPipelineDatadogLogsDestinationType .DATADOG_LOGS)))) .processorGroups( Collections.singletonList( new ObservabilityPipelineConfigProcessorGroup() .enabled(true) .id("my-processor-group") .include("service:my-service") .inputs( Collections.singletonList( "datadog-agent-source")) .processors( Collections.singletonList( new ObservabilityPipelineConfigProcessorItem( new ObservabilityPipelineFilterProcessor() .enabled(true) .id("filter-processor") .include("status:error") .type( ObservabilityPipelineFilterProcessorType .FILTER)))))) .sources( Collections.singletonList( new ObservabilityPipelineConfigSourceItem( new ObservabilityPipelineDatadogAgentSource() .id("datadog-agent-source") .type( ObservabilityPipelineDatadogAgentSourceType .DATADOG_AGENT))))) .name("Main Observability Pipeline")) .type("pipelines")); try { ObservabilityPipeline result = apiInstance.createPipeline(body); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#createPipeline"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` ##### ```java // Create a pipeline with dedupe processor with cache returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; import com.datadog.api.client.v2.model.ObservabilityPipeline; import com.datadog.api.client.v2.model.ObservabilityPipelineConfig; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem; import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType; import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessor; import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorCache; import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorMode; import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorType; import com.datadog.api.client.v2.model.ObservabilityPipelineSpec; import com.datadog.api.client.v2.model.ObservabilityPipelineSpecData; import java.util.Collections; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); ObservabilityPipelineSpec body = new ObservabilityPipelineSpec() .data( new ObservabilityPipelineSpecData() .attributes( new ObservabilityPipelineDataAttributes() .config( new ObservabilityPipelineConfig() .destinations( Collections.singletonList( new ObservabilityPipelineConfigDestinationItem( new ObservabilityPipelineDatadogLogsDestination() .id("datadog-logs-destination") .inputs( Collections.singletonList( "my-processor-group")) .type( ObservabilityPipelineDatadogLogsDestinationType .DATADOG_LOGS)))) .processorGroups( Collections.singletonList( new ObservabilityPipelineConfigProcessorGroup() .enabled(true) .id("my-processor-group") .include("service:my-service") .inputs( Collections.singletonList( "datadog-agent-source")) .processors( Collections.singletonList( new ObservabilityPipelineConfigProcessorItem( new ObservabilityPipelineDedupeProcessor() .enabled(true) .id("dedupe-processor") .include("service:my-service") .type( ObservabilityPipelineDedupeProcessorType .DEDUPE) .fields( Collections.singletonList( "message")) .mode( ObservabilityPipelineDedupeProcessorMode .MATCH) .cache( new ObservabilityPipelineDedupeProcessorCache() .numEvents(5000L))))))) .sources( Collections.singletonList( new ObservabilityPipelineConfigSourceItem( new ObservabilityPipelineDatadogAgentSource() .id("datadog-agent-source") .type( ObservabilityPipelineDatadogAgentSourceType .DATADOG_AGENT))))) .name("Pipeline with Dedupe Cache")) .type("pipelines")); try { ObservabilityPipeline result = apiInstance.createPipeline(body); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#createPipeline"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` ##### ```java // Create a pipeline with dedupe processor without cache returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; import com.datadog.api.client.v2.model.ObservabilityPipeline; import com.datadog.api.client.v2.model.ObservabilityPipelineConfig; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem; import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType; import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessor; import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorMode; import com.datadog.api.client.v2.model.ObservabilityPipelineDedupeProcessorType; import com.datadog.api.client.v2.model.ObservabilityPipelineSpec; import com.datadog.api.client.v2.model.ObservabilityPipelineSpecData; import java.util.Collections; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); ObservabilityPipelineSpec body = new ObservabilityPipelineSpec() .data( new ObservabilityPipelineSpecData() .attributes( new ObservabilityPipelineDataAttributes() .config( new ObservabilityPipelineConfig() .destinations( Collections.singletonList( new ObservabilityPipelineConfigDestinationItem( new ObservabilityPipelineDatadogLogsDestination() .id("datadog-logs-destination") .inputs( Collections.singletonList( "my-processor-group")) .type( ObservabilityPipelineDatadogLogsDestinationType .DATADOG_LOGS)))) .processorGroups( Collections.singletonList( new ObservabilityPipelineConfigProcessorGroup() .enabled(true) .id("my-processor-group") .include("service:my-service") .inputs( Collections.singletonList( "datadog-agent-source")) .processors( Collections.singletonList( new ObservabilityPipelineConfigProcessorItem( new ObservabilityPipelineDedupeProcessor() .enabled(true) .id("dedupe-processor") .include("service:my-service") .type( ObservabilityPipelineDedupeProcessorType .DEDUPE) .fields( Collections.singletonList( "message")) .mode( ObservabilityPipelineDedupeProcessorMode .MATCH)))))) .sources( Collections.singletonList( new ObservabilityPipelineConfigSourceItem( new ObservabilityPipelineDatadogAgentSource() .id("datadog-agent-source") .type( ObservabilityPipelineDatadogAgentSourceType .DATADOG_AGENT))))) .name("Pipeline with Dedupe No Cache")) .type("pipelines")); try { ObservabilityPipeline result = apiInstance.createPipeline(body); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#createPipeline"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=java) and then save the example to `Example.java` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" java "Example.java" ##### ```python """ Create a new pipeline returns "OK" response """ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig from datadog_api_client.v2.model.observability_pipeline_config_processor_group import ( ObservabilityPipelineConfigProcessorGroup, ) from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import ( ObservabilityPipelineDatadogAgentSource, ) from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import ( ObservabilityPipelineDatadogAgentSourceType, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import ( ObservabilityPipelineDatadogLogsDestination, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import ( ObservabilityPipelineDatadogLogsDestinationType, ) from datadog_api_client.v2.model.observability_pipeline_filter_processor import ObservabilityPipelineFilterProcessor from datadog_api_client.v2.model.observability_pipeline_filter_processor_type import ( ObservabilityPipelineFilterProcessorType, ) from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData body = ObservabilityPipelineSpec( data=ObservabilityPipelineSpecData( attributes=ObservabilityPipelineDataAttributes( config=ObservabilityPipelineConfig( destinations=[ ObservabilityPipelineDatadogLogsDestination( id="datadog-logs-destination", inputs=[ "my-processor-group", ], type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS, ), ], processor_groups=[ ObservabilityPipelineConfigProcessorGroup( enabled=True, id="my-processor-group", include="service:my-service", inputs=[ "datadog-agent-source", ], processors=[ ObservabilityPipelineFilterProcessor( enabled=True, id="filter-processor", include="status:error", type=ObservabilityPipelineFilterProcessorType.FILTER, ), ], ), ], sources=[ ObservabilityPipelineDatadogAgentSource( id="datadog-agent-source", type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT, ), ], ), name="Main Observability Pipeline", ), type="pipelines", ), ) configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) response = api_instance.create_pipeline(body=body) print(response) ``` ##### ```python """ Create a pipeline with dedupe processor with cache returns "OK" response """ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig from datadog_api_client.v2.model.observability_pipeline_config_processor_group import ( ObservabilityPipelineConfigProcessorGroup, ) from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import ( ObservabilityPipelineDatadogAgentSource, ) from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import ( ObservabilityPipelineDatadogAgentSourceType, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import ( ObservabilityPipelineDatadogLogsDestination, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import ( ObservabilityPipelineDatadogLogsDestinationType, ) from datadog_api_client.v2.model.observability_pipeline_dedupe_processor import ObservabilityPipelineDedupeProcessor from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_cache import ( ObservabilityPipelineDedupeProcessorCache, ) from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_mode import ( ObservabilityPipelineDedupeProcessorMode, ) from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_type import ( ObservabilityPipelineDedupeProcessorType, ) from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData body = ObservabilityPipelineSpec( data=ObservabilityPipelineSpecData( attributes=ObservabilityPipelineDataAttributes( config=ObservabilityPipelineConfig( destinations=[ ObservabilityPipelineDatadogLogsDestination( id="datadog-logs-destination", inputs=[ "my-processor-group", ], type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS, ), ], processor_groups=[ ObservabilityPipelineConfigProcessorGroup( enabled=True, id="my-processor-group", include="service:my-service", inputs=[ "datadog-agent-source", ], processors=[ ObservabilityPipelineDedupeProcessor( enabled=True, id="dedupe-processor", include="service:my-service", type=ObservabilityPipelineDedupeProcessorType.DEDUPE, fields=[ "message", ], mode=ObservabilityPipelineDedupeProcessorMode.MATCH, cache=ObservabilityPipelineDedupeProcessorCache( num_events=5000, ), ), ], ), ], sources=[ ObservabilityPipelineDatadogAgentSource( id="datadog-agent-source", type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT, ), ], ), name="Pipeline with Dedupe Cache", ), type="pipelines", ), ) configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) response = api_instance.create_pipeline(body=body) print(response) ``` ##### ```python """ Create a pipeline with dedupe processor without cache returns "OK" response """ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig from datadog_api_client.v2.model.observability_pipeline_config_processor_group import ( ObservabilityPipelineConfigProcessorGroup, ) from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import ( ObservabilityPipelineDatadogAgentSource, ) from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import ( ObservabilityPipelineDatadogAgentSourceType, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import ( ObservabilityPipelineDatadogLogsDestination, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import ( ObservabilityPipelineDatadogLogsDestinationType, ) from datadog_api_client.v2.model.observability_pipeline_dedupe_processor import ObservabilityPipelineDedupeProcessor from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_mode import ( ObservabilityPipelineDedupeProcessorMode, ) from datadog_api_client.v2.model.observability_pipeline_dedupe_processor_type import ( ObservabilityPipelineDedupeProcessorType, ) from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData body = ObservabilityPipelineSpec( data=ObservabilityPipelineSpecData( attributes=ObservabilityPipelineDataAttributes( config=ObservabilityPipelineConfig( destinations=[ ObservabilityPipelineDatadogLogsDestination( id="datadog-logs-destination", inputs=[ "my-processor-group", ], type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS, ), ], processor_groups=[ ObservabilityPipelineConfigProcessorGroup( enabled=True, id="my-processor-group", include="service:my-service", inputs=[ "datadog-agent-source", ], processors=[ ObservabilityPipelineDedupeProcessor( enabled=True, id="dedupe-processor", include="service:my-service", type=ObservabilityPipelineDedupeProcessorType.DEDUPE, fields=[ "message", ], mode=ObservabilityPipelineDedupeProcessorMode.MATCH, ), ], ), ], sources=[ ObservabilityPipelineDatadogAgentSource( id="datadog-agent-source", type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT, ), ], ), name="Pipeline with Dedupe No Cache", ), type="pipelines", ), ) configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) response = api_instance.create_pipeline(body=body) print(response) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=python) and then save the example to `example.py` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" python3 "example.py" ##### ```ruby # Create a new pipeline returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({ data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({ attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({ config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({ destinations: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({ id: "datadog-logs-destination", inputs: [ "my-processor-group", ], type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, }), ], processor_groups: [ DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({ enabled: true, id: "my-processor-group", include: "service:my-service", inputs: [ "datadog-agent-source", ], processors: [ DatadogAPIClient::V2::ObservabilityPipelineFilterProcessor.new({ enabled: true, id: "filter-processor", include: "status:error", type: DatadogAPIClient::V2::ObservabilityPipelineFilterProcessorType::FILTER, }), ], }), ], sources: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({ id: "datadog-agent-source", type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, }), ], }), name: "Main Observability Pipeline", }), type: "pipelines", }), }) p api_instance.create_pipeline(body) ``` ##### ```ruby # Create a pipeline with dedupe processor with cache returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({ data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({ attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({ config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({ destinations: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({ id: "datadog-logs-destination", inputs: [ "my-processor-group", ], type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, }), ], processor_groups: [ DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({ enabled: true, id: "my-processor-group", include: "service:my-service", inputs: [ "datadog-agent-source", ], processors: [ DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessor.new({ enabled: true, id: "dedupe-processor", include: "service:my-service", type: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorType::DEDUPE, fields: [ "message", ], mode: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorMode::MATCH, cache: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorCache.new({ num_events: 5000, }), }), ], }), ], sources: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({ id: "datadog-agent-source", type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, }), ], }), name: "Pipeline with Dedupe Cache", }), type: "pipelines", }), }) p api_instance.create_pipeline(body) ``` ##### ```ruby # Create a pipeline with dedupe processor without cache returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({ data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({ attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({ config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({ destinations: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({ id: "datadog-logs-destination", inputs: [ "my-processor-group", ], type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, }), ], processor_groups: [ DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({ enabled: true, id: "my-processor-group", include: "service:my-service", inputs: [ "datadog-agent-source", ], processors: [ DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessor.new({ enabled: true, id: "dedupe-processor", include: "service:my-service", type: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorType::DEDUPE, fields: [ "message", ], mode: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorMode::MATCH, }), ], }), ], sources: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({ id: "datadog-agent-source", type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, }), ], }), name: "Pipeline with Dedupe No Cache", }), type: "pipelines", }), }) p api_instance.create_pipeline(body) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=ruby) and then save the example to `example.rb` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" rb "example.rb" ##### ```rust // Create a new pipeline returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType; use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessor; use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessorType; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpec; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpecData; #[tokio::main] async fn main() { let body = ObservabilityPipelineSpec::new( ObservabilityPipelineSpecData::new( ObservabilityPipelineDataAttributes::new( ObservabilityPipelineConfig::new( vec![ ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination( Box::new( ObservabilityPipelineDatadogLogsDestination::new( "datadog-logs-destination".to_string(), vec!["my-processor-group".to_string()], ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, ), ), ) ], vec![ ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource( Box::new( ObservabilityPipelineDatadogAgentSource::new( "datadog-agent-source".to_string(), ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, ), ), ) ], ).processor_groups( vec![ ObservabilityPipelineConfigProcessorGroup::new( true, "my-processor-group".to_string(), "service:my-service".to_string(), vec!["datadog-agent-source".to_string()], vec![ ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineFilterProcessor( Box::new( ObservabilityPipelineFilterProcessor::new( true, "filter-processor".to_string(), "status:error".to_string(), ObservabilityPipelineFilterProcessorType::FILTER, ), ), ) ], ) ], ), "Main Observability Pipeline".to_string(), ), "pipelines".to_string(), ), ); let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api.create_pipeline(body).await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` ##### ```rust // Create a pipeline with dedupe processor with cache returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType; use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessor; use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorCache; use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorMode; use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorType; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpec; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpecData; #[tokio::main] async fn main() { let body = ObservabilityPipelineSpec::new( ObservabilityPipelineSpecData::new( ObservabilityPipelineDataAttributes::new( ObservabilityPipelineConfig::new( vec![ ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination( Box::new( ObservabilityPipelineDatadogLogsDestination::new( "datadog-logs-destination".to_string(), vec!["my-processor-group".to_string()], ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, ), ), ) ], vec![ ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource( Box::new( ObservabilityPipelineDatadogAgentSource::new( "datadog-agent-source".to_string(), ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, ), ), ) ], ).processor_groups( vec![ ObservabilityPipelineConfigProcessorGroup::new( true, "my-processor-group".to_string(), "service:my-service".to_string(), vec!["datadog-agent-source".to_string()], vec![ ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineDedupeProcessor( Box::new( ObservabilityPipelineDedupeProcessor::new( true, vec!["message".to_string()], "dedupe-processor".to_string(), "service:my-service".to_string(), ObservabilityPipelineDedupeProcessorMode::MATCH, ObservabilityPipelineDedupeProcessorType::DEDUPE, ).cache(ObservabilityPipelineDedupeProcessorCache::new(5000)), ), ) ], ) ], ), "Pipeline with Dedupe Cache".to_string(), ), "pipelines".to_string(), ), ); let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api.create_pipeline(body).await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` ##### ```rust // Create a pipeline with dedupe processor without cache returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType; use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessor; use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorMode; use datadog_api_client::datadogV2::model::ObservabilityPipelineDedupeProcessorType; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpec; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpecData; #[tokio::main] async fn main() { let body = ObservabilityPipelineSpec::new( ObservabilityPipelineSpecData::new( ObservabilityPipelineDataAttributes::new( ObservabilityPipelineConfig::new( vec![ ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination( Box::new( ObservabilityPipelineDatadogLogsDestination::new( "datadog-logs-destination".to_string(), vec!["my-processor-group".to_string()], ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, ), ), ) ], vec![ ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource( Box::new( ObservabilityPipelineDatadogAgentSource::new( "datadog-agent-source".to_string(), ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, ), ), ) ], ).processor_groups( vec![ ObservabilityPipelineConfigProcessorGroup::new( true, "my-processor-group".to_string(), "service:my-service".to_string(), vec!["datadog-agent-source".to_string()], vec![ ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineDedupeProcessor( Box::new( ObservabilityPipelineDedupeProcessor::new( true, vec!["message".to_string()], "dedupe-processor".to_string(), "service:my-service".to_string(), ObservabilityPipelineDedupeProcessorMode::MATCH, ObservabilityPipelineDedupeProcessorType::DEDUPE, ), ), ) ], ) ], ), "Pipeline with Dedupe No Cache".to_string(), ), "pipelines".to_string(), ), ); let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api.create_pipeline(body).await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=rust) and then save the example to `src/main.rs` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" cargo run ##### ```typescript /** * Create a new pipeline returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); const params: v2.ObservabilityPipelinesApiCreatePipelineRequest = { body: { data: { attributes: { config: { destinations: [ { id: "datadog-logs-destination", inputs: ["my-processor-group"], type: "datadog_logs", }, ], processorGroups: [ { enabled: true, id: "my-processor-group", include: "service:my-service", inputs: ["datadog-agent-source"], processors: [ { enabled: true, id: "filter-processor", include: "status:error", type: "filter", }, ], }, ], sources: [ { id: "datadog-agent-source", type: "datadog_agent", }, ], }, name: "Main Observability Pipeline", }, type: "pipelines", }, }, }; apiInstance .createPipeline(params) .then((data: v2.ObservabilityPipeline) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` ##### ```typescript /** * Create a pipeline with dedupe processor with cache returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); const params: v2.ObservabilityPipelinesApiCreatePipelineRequest = { body: { data: { attributes: { config: { destinations: [ { id: "datadog-logs-destination", inputs: ["my-processor-group"], type: "datadog_logs", }, ], processorGroups: [ { enabled: true, id: "my-processor-group", include: "service:my-service", inputs: ["datadog-agent-source"], processors: [ { enabled: true, id: "dedupe-processor", include: "service:my-service", type: "dedupe", fields: ["message"], mode: "match", cache: { numEvents: 5000, }, }, ], }, ], sources: [ { id: "datadog-agent-source", type: "datadog_agent", }, ], }, name: "Pipeline with Dedupe Cache", }, type: "pipelines", }, }, }; apiInstance .createPipeline(params) .then((data: v2.ObservabilityPipeline) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` ##### ```typescript /** * Create a pipeline with dedupe processor without cache returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); const params: v2.ObservabilityPipelinesApiCreatePipelineRequest = { body: { data: { attributes: { config: { destinations: [ { id: "datadog-logs-destination", inputs: ["my-processor-group"], type: "datadog_logs", }, ], processorGroups: [ { enabled: true, id: "my-processor-group", include: "service:my-service", inputs: ["datadog-agent-source"], processors: [ { enabled: true, id: "dedupe-processor", include: "service:my-service", type: "dedupe", fields: ["message"], mode: "match", }, ], }, ], sources: [ { id: "datadog-agent-source", type: "datadog_agent", }, ], }, name: "Pipeline with Dedupe No Cache", }, type: "pipelines", }, }, }; apiInstance .createPipeline(params) .then((data: v2.ObservabilityPipeline) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=typescript) and then save the example to `example.ts` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" tsc "example.ts" {% /tab %} ## Get a specific pipeline{% #get-a-specific-pipeline %} {% tab title="v2" %} | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------ | | ap1.datadoghq.com | GET https://api.ap1.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | ap2.datadoghq.com | GET https://api.ap2.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.datadoghq.eu | GET https://api.datadoghq.eu/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.ddog-gov.com | GET https://api.ddog-gov.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.datadoghq.com | GET https://api.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | us3.datadoghq.com | GET https://api.us3.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | us5.datadoghq.com | GET https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | ### Overview Get a specific pipeline by its ID. This endpoint requires the `observability_pipelines_read` permission. ### Arguments #### Path Parameters | Name | Type | Description | | ----------------------------- | ------ | ----------------------------------- | | pipeline_id [*required*] | string | The ID of the pipeline to retrieve. | ### Response {% tab title="200" %} OK {% tab title="Model" %} Top-level schema representing a pipeline. | Parent field | Field | Type | Description | | -------------------- | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Contains the pipeline's ID, type, and configuration attributes. | | data | attributes [*required*] | object | Defines the pipeline's name and its components (sources, processors, and destinations). | | attributes | config [*required*] | object | Specifies the pipeline's configuration, including its sources, processors, and destinations. | | config | destinations [*required*] | [ ] | A list of destination components where processed logs are sent. | | destinations | Option 1 | object | The `http_client` destination sends data to an HTTP endpoint. | **Supported pipeline types:** logs, metrics | | Option 1 | auth_strategy | enum | HTTP authentication strategy. Allowed enum values: `none,basic,bearer` | | Option 1 | compression | object | Compression configuration for HTTP requests. | | compression | algorithm [*required*] | enum | Compression algorithm. Allowed enum values: `gzip` | | Option 1 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 1 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json` | | Option 1 | id [*required*] | string | The unique identifier for this component. | | Option 1 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 1 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 1 | type [*required*] | enum | The destination type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 1 | uri_key | string | Name of the environment variable or secret that holds the HTTP endpoint URI. | | Option 1 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | destinations | Option 2 | object | The `amazon_opensearch` destination writes logs to Amazon OpenSearch. | **Supported pipeline types:** logs | | Option 2 | auth [*required*] | object | Authentication settings for the Amazon OpenSearch destination. The `strategy` field determines whether basic or AWS-based authentication is used. | | auth | assume_role | string | The ARN of the role to assume (used with `aws` strategy). | | auth | aws_region | string | AWS region | | auth | external_id | string | External ID for the assumed role (used with `aws` strategy). | | auth | session_name | string | Session name for the assumed role (used with `aws` strategy). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | Option 2 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 2 | bulk_index | string | The index to write logs to. | | Option 2 | id [*required*] | string | The unique identifier for this component. | | Option 2 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 2 | type [*required*] | enum | The destination type. The value should always be `amazon_opensearch`. Allowed enum values: `amazon_opensearch` | | destinations | Option 3 | object | The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | bucket [*required*] | string | S3 bucket name. | | Option 3 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 3 | id [*required*] | string | Unique identifier for the destination component. | | Option 3 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 3 | key_prefix | string | Optional prefix for object keys. | | Option 3 | region [*required*] | string | AWS region of the S3 bucket. | | Option 3 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The destination type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | destinations | Option 4 | object | The `amazon_s3_generic` destination sends your logs to an Amazon S3 bucket. | **Supported pipeline types:** logs | | Option 4 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 4 | batch_settings | object | Event batching settings | | batch_settings | batch_size | int64 | Maximum batch size in bytes. | | batch_settings | timeout_secs | int64 | Maximum number of seconds to wait before flushing the batch. | | Option 4 | bucket [*required*] | string | S3 bucket name. | | Option 4 | compression [*required*] | | Compression algorithm applied to encoded logs. | | compression | Option 1 | object | Zstd compression. | | Option 1 | algorithm [*required*] | enum | The compression type. Always `zstd`. Allowed enum values: `zstd` | | Option 1 | level [*required*] | int64 | Zstd compression level. | | compression | Option 2 | object | Gzip compression. | | Option 2 | algorithm [*required*] | enum | The compression type. Always `gzip`. Allowed enum values: `gzip` | | Option 2 | level [*required*] | int64 | Gzip compression level. | | compression | Option 3 | object | Snappy compression. | | Option 3 | algorithm [*required*] | enum | The compression type. Always `snappy`. Allowed enum values: `snappy` | | Option 4 | encoding [*required*] | | Encoding format for the destination. | | encoding | Option 1 | object | JSON encoding. | | Option 1 | type [*required*] | enum | The encoding type. Always `json`. Allowed enum values: `json` | | encoding | Option 2 | object | Parquet encoding. | | Option 2 | type [*required*] | enum | The encoding type. Always `parquet`. Allowed enum values: `parquet` | | Option 4 | id [*required*] | string | Unique identifier for the destination component. | | Option 4 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 4 | key_prefix | string | Optional prefix for object keys. | | Option 4 | region [*required*] | string | AWS region of the S3 bucket. | | Option 4 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 4 | type [*required*] | enum | The destination type. Always `amazon_s3_generic`. Allowed enum values: `amazon_s3_generic` | | destinations | Option 5 | object | The `amazon_security_lake` destination sends your logs to Amazon Security Lake. | **Supported pipeline types:** logs | | Option 5 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 5 | bucket [*required*] | string | Name of the Amazon S3 bucket in Security Lake (3-63 characters). | | Option 5 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 5 | custom_source_name [*required*] | string | Custom source name for the logs in Security Lake. | | Option 5 | id [*required*] | string | Unique identifier for the destination component. | | Option 5 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 5 | region [*required*] | string | AWS region of the S3 bucket. | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The destination type. Always `amazon_security_lake`. Allowed enum values: `amazon_security_lake` | | destinations | Option 6 | object | The `azure_storage` destination forwards logs to an Azure Blob Storage container. | **Supported pipeline types:** logs | | Option 6 | blob_prefix | string | Optional prefix for blobs written to the container. | | Option 6 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 6 | connection_string_key | string | Name of the environment variable or secret that holds the Azure Storage connection string. | | Option 6 | container_name [*required*] | string | The name of the Azure Blob Storage container to store logs in. | | Option 6 | id [*required*] | string | The unique identifier for this component. | | Option 6 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 6 | type [*required*] | enum | The destination type. The value should always be `azure_storage`. Allowed enum values: `azure_storage` | | destinations | Option 7 | object | The `cloud_prem` destination sends logs to Datadog CloudPrem. | **Supported pipeline types:** logs | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the CloudPrem endpoint URL. | | Option 7 | id [*required*] | string | The unique identifier for this component. | | Option 7 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 7 | type [*required*] | enum | The destination type. The value should always be `cloud_prem`. Allowed enum values: `cloud_prem` | | destinations | Option 8 | object | The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM. | **Supported pipeline types:** logs | | Option 8 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 8 | compression | object | Compression configuration for log events. | | compression | algorithm [*required*] | enum | Compression algorithm for log events. Allowed enum values: `gzip,zlib` | | compression | level | int64 | Compression level. | | Option 8 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 8 | endpoint_url_key | string | Name of the environment variable or secret that holds the CrowdStrike endpoint URL. | | Option 8 | id [*required*] | string | The unique identifier for this component. | | Option 8 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | token_key | string | Name of the environment variable or secret that holds the CrowdStrike API token. | | Option 8 | type [*required*] | enum | The destination type. The value should always be `crowdstrike_next_gen_siem`. Allowed enum values: `crowdstrike_next_gen_siem` | | destinations | Option 9 | object | The `datadog_logs` destination forwards logs to Datadog Log Management. | **Supported pipeline types:** logs | | Option 9 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 9 | id [*required*] | string | The unique identifier for this component. | | Option 9 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 9 | routes | [object] | A list of routing rules that forward matching logs to Datadog using dedicated API keys. | | routes | api_key_key | string | Name of the environment variable or secret that stores the Datadog API key used by this route. | | routes | include | string | A Datadog search query that determines which logs are forwarded using this route. | | routes | route_id | string | Unique identifier for this route within the destination. | | routes | site | string | Datadog site where matching logs are sent (for example, `us1`). | | Option 9 | type [*required*] | enum | The destination type. The value should always be `datadog_logs`. Allowed enum values: `datadog_logs` | | destinations | Option 10 | object | The `elasticsearch` destination writes logs to an Elasticsearch cluster. | **Supported pipeline types:** logs | | Option 10 | api_version | enum | The Elasticsearch API version to use. Set to `auto` to auto-detect. Allowed enum values: `auto,v6,v7,v8` | | Option 10 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 10 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 10 | bulk_index | string | The index to write logs to in Elasticsearch. | | Option 10 | data_stream | object | Configuration options for writing to Elasticsearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 10 | endpoint_url_key | string | Name of the environment variable or secret that holds the Elasticsearch endpoint URL. | | Option 10 | id [*required*] | string | The unique identifier for this component. | | Option 10 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 10 | type [*required*] | enum | The destination type. The value should always be `elasticsearch`. Allowed enum values: `elasticsearch` | | destinations | Option 11 | object | The `google_chronicle` destination sends logs to Google Chronicle. | **Supported pipeline types:** logs | | Option 11 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 11 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 11 | customer_id [*required*] | string | The Google Chronicle customer ID. | | Option 11 | encoding | enum | The encoding format for the logs sent to Chronicle. Allowed enum values: `json,raw_message` | | Option 11 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Chronicle endpoint URL. | | Option 11 | id [*required*] | string | The unique identifier for this component. | | Option 11 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 11 | log_type | string | The log type metadata associated with the Chronicle destination. | | Option 11 | type [*required*] | enum | The destination type. The value should always be `google_chronicle`. Allowed enum values: `google_chronicle` | | destinations | Option 12 | object | The `google_cloud_storage` destination stores logs in a Google Cloud Storage (GCS) bucket. It requires a bucket name, Google Cloud authentication, and metadata fields. | **Supported pipeline types:** logs | | Option 12 | acl | enum | Access control list setting for objects written to the bucket. Allowed enum values: `private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control` | | Option 12 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 12 | bucket [*required*] | string | Name of the GCS bucket. | | Option 12 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 12 | id [*required*] | string | Unique identifier for the destination component. | | Option 12 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 12 | key_prefix | string | Optional prefix for object keys within the GCS bucket. | | Option 12 | metadata | [object] | Custom metadata to attach to each object uploaded to the GCS bucket. | | metadata | name [*required*] | string | The metadata key. | | metadata | value [*required*] | string | The metadata value. | | Option 12 | storage_class [*required*] | enum | Storage class used for objects stored in GCS. Allowed enum values: `STANDARD,NEARLINE,COLDLINE,ARCHIVE` | | Option 12 | type [*required*] | enum | The destination type. Always `google_cloud_storage`. Allowed enum values: `google_cloud_storage` | | destinations | Option 13 | object | The `google_pubsub` destination publishes logs to a Google Cloud Pub/Sub topic. | **Supported pipeline types:** logs | | Option 13 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 13 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 13 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 13 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL. | | Option 13 | id [*required*] | string | The unique identifier for this component. | | Option 13 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 13 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub topic. | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | topic [*required*] | string | The Pub/Sub topic name to publish logs to. | | Option 13 | type [*required*] | enum | The destination type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | destinations | Option 14 | object | The `kafka` destination sends logs to Apache Kafka topics. | **Supported pipeline types:** logs | | Option 14 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 14 | compression | enum | Compression codec for Kafka messages. Allowed enum values: `none,gzip,snappy,lz4,zstd` | | Option 14 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 14 | headers_key | string | The field name to use for Kafka message headers. | | Option 14 | id [*required*] | string | The unique identifier for this component. | | Option 14 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 14 | key_field | string | The field name to use as the Kafka message key. | | Option 14 | librdkafka_options | [object] | Optional list of advanced Kafka producer configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 14 | message_timeout_ms | int64 | Maximum time in milliseconds to wait for message delivery confirmation. | | Option 14 | rate_limit_duration_secs | int64 | Duration in seconds for the rate limit window. | | Option 14 | rate_limit_num | int64 | Maximum number of messages allowed per rate limit duration. | | Option 14 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 14 | socket_timeout_ms | int64 | Socket timeout in milliseconds for network requests. | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | topic [*required*] | string | The Kafka topic name to publish logs to. | | Option 14 | type [*required*] | enum | The destination type. The value should always be `kafka`. Allowed enum values: `kafka` | | destinations | Option 15 | object | The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel. | **Supported pipeline types:** logs | | Option 15 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 15 | client_id [*required*] | string | Azure AD client ID used for authentication. | | Option 15 | client_secret_key | string | Name of the environment variable or secret that holds the Azure AD client secret. | | Option 15 | dce_uri_key | string | Name of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI. | | Option 15 | dcr_immutable_id [*required*] | string | The immutable ID of the Data Collection Rule (DCR). | | Option 15 | id [*required*] | string | The unique identifier for this component. | | Option 15 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 15 | table [*required*] | string | The name of the Log Analytics table where logs are sent. | | Option 15 | tenant_id [*required*] | string | Azure AD tenant ID. | | Option 15 | type [*required*] | enum | The destination type. The value should always be `microsoft_sentinel`. Allowed enum values: `microsoft_sentinel` | | destinations | Option 16 | object | The `new_relic` destination sends logs to the New Relic platform. | **Supported pipeline types:** logs | | Option 16 | account_id_key | string | Name of the environment variable or secret that holds the New Relic account ID. | | Option 16 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 16 | id [*required*] | string | The unique identifier for this component. | | Option 16 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 16 | license_key_key | string | Name of the environment variable or secret that holds the New Relic license key. | | Option 16 | region [*required*] | enum | The New Relic region. Allowed enum values: `us,eu` | | Option 16 | type [*required*] | enum | The destination type. The value should always be `new_relic`. Allowed enum values: `new_relic` | | destinations | Option 17 | object | The `opensearch` destination writes logs to an OpenSearch cluster. | **Supported pipeline types:** logs | | Option 17 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 17 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 17 | bulk_index | string | The index to write logs to. | | Option 17 | data_stream | object | Configuration options for writing to OpenSearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 17 | endpoint_url_key | string | Name of the environment variable or secret that holds the OpenSearch endpoint URL. | | Option 17 | id [*required*] | string | The unique identifier for this component. | | Option 17 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 17 | type [*required*] | enum | The destination type. The value should always be `opensearch`. Allowed enum values: `opensearch` | | destinations | Option 18 | object | The `rsyslog` destination forwards logs to an external `rsyslog` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 18 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 18 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog server endpoint URL. | | Option 18 | id [*required*] | string | The unique identifier for this component. | | Option 18 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 18 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 18 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 18 | type [*required*] | enum | The destination type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | destinations | Option 19 | object | The `sentinel_one` destination sends logs to SentinelOne. | **Supported pipeline types:** logs | | Option 19 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 19 | id [*required*] | string | The unique identifier for this component. | | Option 19 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 19 | region [*required*] | enum | The SentinelOne region to send logs to. Allowed enum values: `us,eu,ca,data_set_us` | | Option 19 | token_key | string | Name of the environment variable or secret that holds the SentinelOne API token. | | Option 19 | type [*required*] | enum | The destination type. The value should always be `sentinel_one`. Allowed enum values: `sentinel_one` | | destinations | Option 20 | object | The `socket` destination sends logs over TCP or UDP to a remote server. | **Supported pipeline types:** logs | | Option 20 | address_key | string | Name of the environment variable or secret that holds the socket address (host:port). | | Option 20 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 20 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 20 | framing [*required*] | | Framing method configuration. | | framing | Option 1 | object | Each log event is delimited by a newline character. | | Option 1 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod` object. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Event data is not delimited at all. | | Option 2 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod` object. Allowed enum values: `bytes` | | framing | Option 3 | object | Each log event is separated using the specified delimiter character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used as a delimiter. | | Option 3 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod` object. Allowed enum values: `character_delimited` | | Option 20 | id [*required*] | string | The unique identifier for this component. | | Option 20 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 20 | mode [*required*] | enum | Protocol used to send logs. Allowed enum values: `tcp,udp` | | Option 20 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 20 | type [*required*] | enum | The destination type. The value should always be `socket`. Allowed enum values: `socket` | | destinations | Option 21 | object | The `splunk_hec` destination forwards logs to Splunk using the HTTP Event Collector (HEC). | **Supported pipeline types:** logs | | Option 21 | auto_extract_timestamp | boolean | If `true`, Splunk tries to extract timestamps from incoming log events. If `false`, Splunk assigns the time the event was received. | | Option 21 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 21 | encoding | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 21 | endpoint_url_key | string | Name of the environment variable or secret that holds the Splunk HEC endpoint URL. | | Option 21 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 21 | index | string | Optional name of the Splunk index where logs are written. | | Option 21 | indexed_fields | [string] | List of log field names to send as indexed fields to Splunk HEC. Available only when `encoding` is `json`. | | Option 21 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 21 | sourcetype | string | The Splunk sourcetype to assign to log events. | | Option 21 | token_key | string | Name of the environment variable or secret that holds the Splunk HEC token. | | Option 21 | type [*required*] | enum | The destination type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | destinations | Option 22 | object | The `sumo_logic` destination forwards logs to Sumo Logic. | **Supported pipeline types:** logs | | Option 22 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 22 | encoding | enum | The output encoding format. Allowed enum values: `json,raw_message,logfmt` | | Option 22 | endpoint_url_key | string | Name of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL. | | Option 22 | header_custom_fields | [object] | A list of custom headers to include in the request to Sumo Logic. | | header_custom_fields | name [*required*] | string | The header field name. | | header_custom_fields | value [*required*] | string | The header field value. | | Option 22 | header_host_name | string | Optional override for the host name header. | | Option 22 | header_source_category | string | Optional override for the source category header. | | Option 22 | header_source_name | string | Optional override for the source name header. | | Option 22 | id [*required*] | string | The unique identifier for this component. | | Option 22 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 22 | type [*required*] | enum | The destination type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | destinations | Option 23 | object | The `syslog_ng` destination forwards logs to an external `syslog-ng` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 23 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 23 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog-ng server endpoint URL. | | Option 23 | id [*required*] | string | The unique identifier for this component. | | Option 23 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 23 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 23 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 23 | type [*required*] | enum | The destination type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | destinations | Option 24 | object | The `datadog_metrics` destination forwards metrics to Datadog. | **Supported pipeline types:** metrics | | Option 24 | id [*required*] | string | The unique identifier for this component. | | Option 24 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 24 | type [*required*] | enum | The destination type. The value should always be `datadog_metrics`. Allowed enum values: `datadog_metrics` | | config | pipeline_type | enum | The type of data being ingested. Defaults to `logs` if not specified. Allowed enum values: `logs,metrics` | | config | processor_groups | [object] | A list of processor groups that transform or enrich log data. | | processor_groups | display_name | string | The display name for a component. | | processor_groups | enabled [*required*] | boolean | Whether this processor group is enabled. | | processor_groups | id [*required*] | string | The unique identifier for the processor group. | | processor_groups | include [*required*] | string | Conditional expression for when this processor group should execute. | | processor_groups | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processor_groups | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | processors | [object] | **DEPRECATED**: A list of processor groups that transform or enrich log data. | **Deprecated:** This field is deprecated, you should now use the processor_groups field. | | processors | display_name | string | The display name for a component. | | processors | enabled [*required*] | boolean | Whether this processor group is enabled. | | processors | id [*required*] | string | The unique identifier for the processor group. | | processors | include [*required*] | string | Conditional expression for when this processor group should execute. | | processors | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processors | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | sources [*required*] | [ ] | A list of configured data sources for the pipeline. | | sources | Option 1 | object | The `datadog_agent` source collects logs/metrics from the Datadog Agent. | **Supported pipeline types:** logs, metrics | | Option 1 | address_key | string | Name of the environment variable or secret that holds the listen address for the Datadog Agent source. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | type [*required*] | enum | The source type. The value should always be `datadog_agent`. Allowed enum values: `datadog_agent` | | sources | Option 2 | object | The `amazon_data_firehose` source ingests logs from AWS Data Firehose. | **Supported pipeline types:** logs | | Option 2 | address_key | string | Name of the environment variable or secret that holds the Firehose delivery stream address. | | Option 2 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 2 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 2 | type [*required*] | enum | The source type. The value should always be `amazon_data_firehose`. Allowed enum values: `amazon_data_firehose` | | sources | Option 3 | object | The `amazon_s3` source ingests logs from an Amazon S3 bucket. It supports AWS authentication and TLS encryption. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | region [*required*] | string | AWS region where the S3 bucket resides. | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The source type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | Option 3 | url_key | string | Name of the environment variable or secret that holds the S3 bucket URL. | | sources | Option 4 | object | The `fluent_bit` source ingests logs from Fluent Bit. | **Supported pipeline types:** logs | | Option 4 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent Bit receiver. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 4 | type [*required*] | enum | The source type. The value should always be `fluent_bit`. Allowed enum values: `fluent_bit` | | sources | Option 5 | object | The `fluentd` source ingests logs from a Fluentd-compatible service. | **Supported pipeline types:** logs | | Option 5 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent receiver. | | Option 5 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The source type. The value should always be `fluentd. Allowed enum values: `fluentd` | | sources | Option 6 | object | The `google_pubsub` source ingests logs from a Google Cloud Pub/Sub subscription. | **Supported pipeline types:** logs | | Option 6 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 6 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub subscription. | | Option 6 | subscription [*required*] | string | The Pub/Sub subscription name from which messages are consumed. | | Option 6 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 6 | type [*required*] | enum | The source type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | sources | Option 7 | object | The `http_client` source scrapes logs from HTTP endpoints at regular intervals. | **Supported pipeline types:** logs | | Option 7 | auth_strategy | enum | Optional authentication strategy for HTTP requests. Allowed enum values: `none,basic,bearer,custom` | | Option 7 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 7 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the HTTP endpoint URL to scrape. | | Option 7 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 7 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 7 | scrape_interval_secs | int64 | The interval (in seconds) between HTTP scrape requests. | | Option 7 | scrape_timeout_secs | int64 | The timeout (in seconds) for each scrape request. | | Option 7 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 7 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 7 | type [*required*] | enum | The source type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 7 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | sources | Option 8 | object | The `http_server` source collects logs over HTTP POST from external services. | **Supported pipeline types:** logs | | Option 8 | address_key | string | Name of the environment variable or secret that holds the listen address for the HTTP server. | | Option 8 | auth_strategy [*required*] | enum | HTTP authentication method. Allowed enum values: `none,plain` | | Option 8 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 8 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 8 | id [*required*] | string | Unique ID for the HTTP server source. | | Option 8 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `plain`). | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | type [*required*] | enum | The source type. The value should always be `http_server`. Allowed enum values: `http_server` | | Option 8 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `plain`). | | sources | Option 9 | object | The `kafka` source ingests data from Apache Kafka topics. | **Supported pipeline types:** logs | | Option 9 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 9 | group_id [*required*] | string | Consumer group ID used by the Kafka client. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 9 | librdkafka_options | [object] | Optional list of advanced Kafka client configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 9 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 9 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 9 | topics [*required*] | [string] | A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified. | | Option 9 | type [*required*] | enum | The source type. The value should always be `kafka`. Allowed enum values: `kafka` | | sources | Option 10 | object | The `logstash` source ingests logs from a Logstash forwarder. | **Supported pipeline types:** logs | | Option 10 | address_key | string | Name of the environment variable or secret that holds the listen address for the Logstash receiver. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 10 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 10 | type [*required*] | enum | The source type. The value should always be `logstash`. Allowed enum values: `logstash` | | sources | Option 11 | object | The `rsyslog` source listens for logs over TCP or UDP from an `rsyslog` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 11 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog receiver. | | Option 11 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 11 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 11 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 11 | type [*required*] | enum | The source type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | sources | Option 12 | object | The `socket` source ingests logs over TCP or UDP. | **Supported pipeline types:** logs | | Option 12 | address_key | string | Name of the environment variable or secret that holds the listen address for the socket. | | Option 12 | framing [*required*] | | Framing method configuration for the socket source. | | framing | Option 1 | object | Byte frames which are delimited by a newline character. | | Option 1 | method [*required*] | enum | Byte frames which are delimited by a newline character. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). | | Option 2 | method [*required*] | enum | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). Allowed enum values: `bytes` | | framing | Option 3 | object | Byte frames which are delimited by a chosen character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used to delimit events. | | Option 3 | method [*required*] | enum | Byte frames which are delimited by a chosen character. Allowed enum values: `character_delimited` | | framing | Option 4 | object | Byte frames according to the octet counting format as per RFC6587. | | Option 4 | method [*required*] | enum | Byte frames according to the octet counting format as per RFC6587. Allowed enum values: `octet_counting` | | framing | Option 5 | object | Byte frames which are chunked GELF messages. | | Option 5 | method [*required*] | enum | Byte frames which are chunked GELF messages. Allowed enum values: `chunked_gelf` | | Option 12 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 12 | mode [*required*] | enum | Protocol used to receive logs. Allowed enum values: `tcp,udp` | | Option 12 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 12 | type [*required*] | enum | The source type. The value should always be `socket`. Allowed enum values: `socket` | | sources | Option 13 | object | The `splunk_hec` source implements the Splunk HTTP Event Collector (HEC) API. | **Supported pipeline types:** logs | | Option 13 | address_key | string | Name of the environment variable or secret that holds the listen address for the HEC API. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | type [*required*] | enum | The source type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | sources | Option 14 | object | The `splunk_tcp` source receives logs from a Splunk Universal Forwarder over TCP. TLS is supported for secure transmission. | **Supported pipeline types:** logs | | Option 14 | address_key | string | Name of the environment variable or secret that holds the listen address for the Splunk TCP receiver. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | type [*required*] | enum | The source type. Always `splunk_tcp`. Allowed enum values: `splunk_tcp` | | sources | Option 15 | object | The `sumo_logic` source receives logs from Sumo Logic collectors. | **Supported pipeline types:** logs | | Option 15 | address_key | string | Name of the environment variable or secret that holds the listen address for the Sumo Logic receiver. | | Option 15 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 15 | type [*required*] | enum | The source type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | sources | Option 16 | object | The `syslog_ng` source listens for logs over TCP or UDP from a `syslog-ng` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 16 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog-ng receiver. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 16 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 16 | type [*required*] | enum | The source type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | sources | Option 17 | object | The `opentelemetry` source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP. | **Supported pipeline types:** logs, metrics | | Option 17 | grpc_address_key | string | Environment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | http_address_key | string | Environment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 17 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 17 | type [*required*] | enum | The source type. The value should always be `opentelemetry`. Allowed enum values: `opentelemetry` | | config | use_legacy_search_syntax | boolean | Set to `true` to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to `false`. The legacy syntax is deprecated and will eventually be removed. Requires Observability Pipelines Worker 2.11 or later. See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax/) for more information. | | attributes | name [*required*] | string | Name of the pipeline. | | data | id [*required*] | string | Unique identifier for the pipeline. | | data | type [*required*] | string | The resource type identifier. For pipeline resources, this should always be set to `pipelines`. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": { "destinations": [ { "auth_strategy": "basic", "compression": { "algorithm": "gzip" }, "custom_key": "HTTP_AUTH_CUSTOM_HEADER", "encoding": "json", "id": "http-client-destination", "inputs": [ "filter-processor" ], "password_key": "HTTP_AUTH_PASSWORD", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "token_key": "HTTP_AUTH_TOKEN", "type": "http_client", "uri_key": "HTTP_DESTINATION_URI", "username_key": "HTTP_AUTH_USERNAME" } ], "pipeline_type": "logs", "processor_groups": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "processors": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "sources": [ { "address_key": "DATADOG_AGENT_ADDRESS", "id": "datadog-agent-source", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "type": "datadog_agent" } ], "use_legacy_search_syntax": false }, "name": "Main Observability Pipeline" }, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines" } } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport pipeline_id="CHANGE_ME"\# Curl commandcurl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/${pipeline_id}" \ -H "Accept: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" ##### ```python """ Get a specific pipeline returns "OK" response """ from os import environ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi # there is a valid "pipeline" in the system PIPELINE_DATA_ID = environ["PIPELINE_DATA_ID"] configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) response = api_instance.get_pipeline( pipeline_id=PIPELINE_DATA_ID, ) print(response) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=python) and then save the example to `example.py` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" python3 "example.py" ##### ```ruby # Get a specific pipeline returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new # there is a valid "pipeline" in the system PIPELINE_DATA_ID = ENV["PIPELINE_DATA_ID"] p api_instance.get_pipeline(PIPELINE_DATA_ID) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=ruby) and then save the example to `example.rb` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" rb "example.rb" ##### ```go // Get a specific pipeline returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { // there is a valid "pipeline" in the system PipelineDataID := os.Getenv("PIPELINE_DATA_ID") ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) resp, r, err := api.GetPipeline(ctx, PipelineDataID) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.GetPipeline`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.GetPipeline`:\n%s\n", responseContent) } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=go) and then save the example to `main.go` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" go run "main.go" ##### ```java // Get a specific pipeline returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; import com.datadog.api.client.v2.model.ObservabilityPipeline; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); // there is a valid "pipeline" in the system String PIPELINE_DATA_ID = System.getenv("PIPELINE_DATA_ID"); try { ObservabilityPipeline result = apiInstance.getPipeline(PIPELINE_DATA_ID); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#getPipeline"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=java) and then save the example to `Example.java` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" java "Example.java" ##### ```rust // Get a specific pipeline returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; #[tokio::main] async fn main() { // there is a valid "pipeline" in the system let pipeline_data_id = std::env::var("PIPELINE_DATA_ID").unwrap(); let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api.get_pipeline(pipeline_data_id.clone()).await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=rust) and then save the example to `src/main.rs` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" cargo run ##### ```typescript /** * Get a specific pipeline returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); // there is a valid "pipeline" in the system const PIPELINE_DATA_ID = process.env.PIPELINE_DATA_ID as string; const params: v2.ObservabilityPipelinesApiGetPipelineRequest = { pipelineId: PIPELINE_DATA_ID, }; apiInstance .getPipeline(params) .then((data: v2.ObservabilityPipeline) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=typescript) and then save the example to `example.ts` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" tsc "example.ts" {% /tab %} ## Update a pipeline{% #update-a-pipeline %} {% tab title="v2" %} | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------ | | ap1.datadoghq.com | PUT https://api.ap1.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | ap2.datadoghq.com | PUT https://api.ap2.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.datadoghq.eu | PUT https://api.datadoghq.eu/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.ddog-gov.com | PUT https://api.ddog-gov.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.datadoghq.com | PUT https://api.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | us3.datadoghq.com | PUT https://api.us3.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | us5.datadoghq.com | PUT https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | ### Overview Update a pipeline. This endpoint requires the `observability_pipelines_deploy` permission. ### Arguments #### Path Parameters | Name | Type | Description | | ----------------------------- | ------ | --------------------------------- | | pipeline_id [*required*] | string | The ID of the pipeline to update. | ### Request #### Body Data (required) {% tab title="Model" %} | Parent field | Field | Type | Description | | -------------------- | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Contains the pipeline's ID, type, and configuration attributes. | | data | attributes [*required*] | object | Defines the pipeline's name and its components (sources, processors, and destinations). | | attributes | config [*required*] | object | Specifies the pipeline's configuration, including its sources, processors, and destinations. | | config | destinations [*required*] | [ ] | A list of destination components where processed logs are sent. | | destinations | Option 1 | object | The `http_client` destination sends data to an HTTP endpoint. | **Supported pipeline types:** logs, metrics | | Option 1 | auth_strategy | enum | HTTP authentication strategy. Allowed enum values: `none,basic,bearer` | | Option 1 | compression | object | Compression configuration for HTTP requests. | | compression | algorithm [*required*] | enum | Compression algorithm. Allowed enum values: `gzip` | | Option 1 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 1 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json` | | Option 1 | id [*required*] | string | The unique identifier for this component. | | Option 1 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 1 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 1 | type [*required*] | enum | The destination type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 1 | uri_key | string | Name of the environment variable or secret that holds the HTTP endpoint URI. | | Option 1 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | destinations | Option 2 | object | The `amazon_opensearch` destination writes logs to Amazon OpenSearch. | **Supported pipeline types:** logs | | Option 2 | auth [*required*] | object | Authentication settings for the Amazon OpenSearch destination. The `strategy` field determines whether basic or AWS-based authentication is used. | | auth | assume_role | string | The ARN of the role to assume (used with `aws` strategy). | | auth | aws_region | string | AWS region | | auth | external_id | string | External ID for the assumed role (used with `aws` strategy). | | auth | session_name | string | Session name for the assumed role (used with `aws` strategy). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | Option 2 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 2 | bulk_index | string | The index to write logs to. | | Option 2 | id [*required*] | string | The unique identifier for this component. | | Option 2 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 2 | type [*required*] | enum | The destination type. The value should always be `amazon_opensearch`. Allowed enum values: `amazon_opensearch` | | destinations | Option 3 | object | The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | bucket [*required*] | string | S3 bucket name. | | Option 3 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 3 | id [*required*] | string | Unique identifier for the destination component. | | Option 3 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 3 | key_prefix | string | Optional prefix for object keys. | | Option 3 | region [*required*] | string | AWS region of the S3 bucket. | | Option 3 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The destination type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | destinations | Option 4 | object | The `amazon_s3_generic` destination sends your logs to an Amazon S3 bucket. | **Supported pipeline types:** logs | | Option 4 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 4 | batch_settings | object | Event batching settings | | batch_settings | batch_size | int64 | Maximum batch size in bytes. | | batch_settings | timeout_secs | int64 | Maximum number of seconds to wait before flushing the batch. | | Option 4 | bucket [*required*] | string | S3 bucket name. | | Option 4 | compression [*required*] | | Compression algorithm applied to encoded logs. | | compression | Option 1 | object | Zstd compression. | | Option 1 | algorithm [*required*] | enum | The compression type. Always `zstd`. Allowed enum values: `zstd` | | Option 1 | level [*required*] | int64 | Zstd compression level. | | compression | Option 2 | object | Gzip compression. | | Option 2 | algorithm [*required*] | enum | The compression type. Always `gzip`. Allowed enum values: `gzip` | | Option 2 | level [*required*] | int64 | Gzip compression level. | | compression | Option 3 | object | Snappy compression. | | Option 3 | algorithm [*required*] | enum | The compression type. Always `snappy`. Allowed enum values: `snappy` | | Option 4 | encoding [*required*] | | Encoding format for the destination. | | encoding | Option 1 | object | JSON encoding. | | Option 1 | type [*required*] | enum | The encoding type. Always `json`. Allowed enum values: `json` | | encoding | Option 2 | object | Parquet encoding. | | Option 2 | type [*required*] | enum | The encoding type. Always `parquet`. Allowed enum values: `parquet` | | Option 4 | id [*required*] | string | Unique identifier for the destination component. | | Option 4 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 4 | key_prefix | string | Optional prefix for object keys. | | Option 4 | region [*required*] | string | AWS region of the S3 bucket. | | Option 4 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 4 | type [*required*] | enum | The destination type. Always `amazon_s3_generic`. Allowed enum values: `amazon_s3_generic` | | destinations | Option 5 | object | The `amazon_security_lake` destination sends your logs to Amazon Security Lake. | **Supported pipeline types:** logs | | Option 5 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 5 | bucket [*required*] | string | Name of the Amazon S3 bucket in Security Lake (3-63 characters). | | Option 5 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 5 | custom_source_name [*required*] | string | Custom source name for the logs in Security Lake. | | Option 5 | id [*required*] | string | Unique identifier for the destination component. | | Option 5 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 5 | region [*required*] | string | AWS region of the S3 bucket. | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The destination type. Always `amazon_security_lake`. Allowed enum values: `amazon_security_lake` | | destinations | Option 6 | object | The `azure_storage` destination forwards logs to an Azure Blob Storage container. | **Supported pipeline types:** logs | | Option 6 | blob_prefix | string | Optional prefix for blobs written to the container. | | Option 6 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 6 | connection_string_key | string | Name of the environment variable or secret that holds the Azure Storage connection string. | | Option 6 | container_name [*required*] | string | The name of the Azure Blob Storage container to store logs in. | | Option 6 | id [*required*] | string | The unique identifier for this component. | | Option 6 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 6 | type [*required*] | enum | The destination type. The value should always be `azure_storage`. Allowed enum values: `azure_storage` | | destinations | Option 7 | object | The `cloud_prem` destination sends logs to Datadog CloudPrem. | **Supported pipeline types:** logs | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the CloudPrem endpoint URL. | | Option 7 | id [*required*] | string | The unique identifier for this component. | | Option 7 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 7 | type [*required*] | enum | The destination type. The value should always be `cloud_prem`. Allowed enum values: `cloud_prem` | | destinations | Option 8 | object | The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM. | **Supported pipeline types:** logs | | Option 8 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 8 | compression | object | Compression configuration for log events. | | compression | algorithm [*required*] | enum | Compression algorithm for log events. Allowed enum values: `gzip,zlib` | | compression | level | int64 | Compression level. | | Option 8 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 8 | endpoint_url_key | string | Name of the environment variable or secret that holds the CrowdStrike endpoint URL. | | Option 8 | id [*required*] | string | The unique identifier for this component. | | Option 8 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | token_key | string | Name of the environment variable or secret that holds the CrowdStrike API token. | | Option 8 | type [*required*] | enum | The destination type. The value should always be `crowdstrike_next_gen_siem`. Allowed enum values: `crowdstrike_next_gen_siem` | | destinations | Option 9 | object | The `datadog_logs` destination forwards logs to Datadog Log Management. | **Supported pipeline types:** logs | | Option 9 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 9 | id [*required*] | string | The unique identifier for this component. | | Option 9 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 9 | routes | [object] | A list of routing rules that forward matching logs to Datadog using dedicated API keys. | | routes | api_key_key | string | Name of the environment variable or secret that stores the Datadog API key used by this route. | | routes | include | string | A Datadog search query that determines which logs are forwarded using this route. | | routes | route_id | string | Unique identifier for this route within the destination. | | routes | site | string | Datadog site where matching logs are sent (for example, `us1`). | | Option 9 | type [*required*] | enum | The destination type. The value should always be `datadog_logs`. Allowed enum values: `datadog_logs` | | destinations | Option 10 | object | The `elasticsearch` destination writes logs to an Elasticsearch cluster. | **Supported pipeline types:** logs | | Option 10 | api_version | enum | The Elasticsearch API version to use. Set to `auto` to auto-detect. Allowed enum values: `auto,v6,v7,v8` | | Option 10 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 10 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 10 | bulk_index | string | The index to write logs to in Elasticsearch. | | Option 10 | data_stream | object | Configuration options for writing to Elasticsearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 10 | endpoint_url_key | string | Name of the environment variable or secret that holds the Elasticsearch endpoint URL. | | Option 10 | id [*required*] | string | The unique identifier for this component. | | Option 10 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 10 | type [*required*] | enum | The destination type. The value should always be `elasticsearch`. Allowed enum values: `elasticsearch` | | destinations | Option 11 | object | The `google_chronicle` destination sends logs to Google Chronicle. | **Supported pipeline types:** logs | | Option 11 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 11 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 11 | customer_id [*required*] | string | The Google Chronicle customer ID. | | Option 11 | encoding | enum | The encoding format for the logs sent to Chronicle. Allowed enum values: `json,raw_message` | | Option 11 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Chronicle endpoint URL. | | Option 11 | id [*required*] | string | The unique identifier for this component. | | Option 11 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 11 | log_type | string | The log type metadata associated with the Chronicle destination. | | Option 11 | type [*required*] | enum | The destination type. The value should always be `google_chronicle`. Allowed enum values: `google_chronicle` | | destinations | Option 12 | object | The `google_cloud_storage` destination stores logs in a Google Cloud Storage (GCS) bucket. It requires a bucket name, Google Cloud authentication, and metadata fields. | **Supported pipeline types:** logs | | Option 12 | acl | enum | Access control list setting for objects written to the bucket. Allowed enum values: `private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control` | | Option 12 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 12 | bucket [*required*] | string | Name of the GCS bucket. | | Option 12 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 12 | id [*required*] | string | Unique identifier for the destination component. | | Option 12 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 12 | key_prefix | string | Optional prefix for object keys within the GCS bucket. | | Option 12 | metadata | [object] | Custom metadata to attach to each object uploaded to the GCS bucket. | | metadata | name [*required*] | string | The metadata key. | | metadata | value [*required*] | string | The metadata value. | | Option 12 | storage_class [*required*] | enum | Storage class used for objects stored in GCS. Allowed enum values: `STANDARD,NEARLINE,COLDLINE,ARCHIVE` | | Option 12 | type [*required*] | enum | The destination type. Always `google_cloud_storage`. Allowed enum values: `google_cloud_storage` | | destinations | Option 13 | object | The `google_pubsub` destination publishes logs to a Google Cloud Pub/Sub topic. | **Supported pipeline types:** logs | | Option 13 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 13 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 13 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 13 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL. | | Option 13 | id [*required*] | string | The unique identifier for this component. | | Option 13 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 13 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub topic. | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | topic [*required*] | string | The Pub/Sub topic name to publish logs to. | | Option 13 | type [*required*] | enum | The destination type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | destinations | Option 14 | object | The `kafka` destination sends logs to Apache Kafka topics. | **Supported pipeline types:** logs | | Option 14 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 14 | compression | enum | Compression codec for Kafka messages. Allowed enum values: `none,gzip,snappy,lz4,zstd` | | Option 14 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 14 | headers_key | string | The field name to use for Kafka message headers. | | Option 14 | id [*required*] | string | The unique identifier for this component. | | Option 14 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 14 | key_field | string | The field name to use as the Kafka message key. | | Option 14 | librdkafka_options | [object] | Optional list of advanced Kafka producer configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 14 | message_timeout_ms | int64 | Maximum time in milliseconds to wait for message delivery confirmation. | | Option 14 | rate_limit_duration_secs | int64 | Duration in seconds for the rate limit window. | | Option 14 | rate_limit_num | int64 | Maximum number of messages allowed per rate limit duration. | | Option 14 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 14 | socket_timeout_ms | int64 | Socket timeout in milliseconds for network requests. | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | topic [*required*] | string | The Kafka topic name to publish logs to. | | Option 14 | type [*required*] | enum | The destination type. The value should always be `kafka`. Allowed enum values: `kafka` | | destinations | Option 15 | object | The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel. | **Supported pipeline types:** logs | | Option 15 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 15 | client_id [*required*] | string | Azure AD client ID used for authentication. | | Option 15 | client_secret_key | string | Name of the environment variable or secret that holds the Azure AD client secret. | | Option 15 | dce_uri_key | string | Name of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI. | | Option 15 | dcr_immutable_id [*required*] | string | The immutable ID of the Data Collection Rule (DCR). | | Option 15 | id [*required*] | string | The unique identifier for this component. | | Option 15 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 15 | table [*required*] | string | The name of the Log Analytics table where logs are sent. | | Option 15 | tenant_id [*required*] | string | Azure AD tenant ID. | | Option 15 | type [*required*] | enum | The destination type. The value should always be `microsoft_sentinel`. Allowed enum values: `microsoft_sentinel` | | destinations | Option 16 | object | The `new_relic` destination sends logs to the New Relic platform. | **Supported pipeline types:** logs | | Option 16 | account_id_key | string | Name of the environment variable or secret that holds the New Relic account ID. | | Option 16 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 16 | id [*required*] | string | The unique identifier for this component. | | Option 16 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 16 | license_key_key | string | Name of the environment variable or secret that holds the New Relic license key. | | Option 16 | region [*required*] | enum | The New Relic region. Allowed enum values: `us,eu` | | Option 16 | type [*required*] | enum | The destination type. The value should always be `new_relic`. Allowed enum values: `new_relic` | | destinations | Option 17 | object | The `opensearch` destination writes logs to an OpenSearch cluster. | **Supported pipeline types:** logs | | Option 17 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 17 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 17 | bulk_index | string | The index to write logs to. | | Option 17 | data_stream | object | Configuration options for writing to OpenSearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 17 | endpoint_url_key | string | Name of the environment variable or secret that holds the OpenSearch endpoint URL. | | Option 17 | id [*required*] | string | The unique identifier for this component. | | Option 17 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 17 | type [*required*] | enum | The destination type. The value should always be `opensearch`. Allowed enum values: `opensearch` | | destinations | Option 18 | object | The `rsyslog` destination forwards logs to an external `rsyslog` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 18 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 18 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog server endpoint URL. | | Option 18 | id [*required*] | string | The unique identifier for this component. | | Option 18 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 18 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 18 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 18 | type [*required*] | enum | The destination type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | destinations | Option 19 | object | The `sentinel_one` destination sends logs to SentinelOne. | **Supported pipeline types:** logs | | Option 19 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 19 | id [*required*] | string | The unique identifier for this component. | | Option 19 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 19 | region [*required*] | enum | The SentinelOne region to send logs to. Allowed enum values: `us,eu,ca,data_set_us` | | Option 19 | token_key | string | Name of the environment variable or secret that holds the SentinelOne API token. | | Option 19 | type [*required*] | enum | The destination type. The value should always be `sentinel_one`. Allowed enum values: `sentinel_one` | | destinations | Option 20 | object | The `socket` destination sends logs over TCP or UDP to a remote server. | **Supported pipeline types:** logs | | Option 20 | address_key | string | Name of the environment variable or secret that holds the socket address (host:port). | | Option 20 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 20 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 20 | framing [*required*] | | Framing method configuration. | | framing | Option 1 | object | Each log event is delimited by a newline character. | | Option 1 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod` object. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Event data is not delimited at all. | | Option 2 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod` object. Allowed enum values: `bytes` | | framing | Option 3 | object | Each log event is separated using the specified delimiter character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used as a delimiter. | | Option 3 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod` object. Allowed enum values: `character_delimited` | | Option 20 | id [*required*] | string | The unique identifier for this component. | | Option 20 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 20 | mode [*required*] | enum | Protocol used to send logs. Allowed enum values: `tcp,udp` | | Option 20 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 20 | type [*required*] | enum | The destination type. The value should always be `socket`. Allowed enum values: `socket` | | destinations | Option 21 | object | The `splunk_hec` destination forwards logs to Splunk using the HTTP Event Collector (HEC). | **Supported pipeline types:** logs | | Option 21 | auto_extract_timestamp | boolean | If `true`, Splunk tries to extract timestamps from incoming log events. If `false`, Splunk assigns the time the event was received. | | Option 21 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 21 | encoding | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 21 | endpoint_url_key | string | Name of the environment variable or secret that holds the Splunk HEC endpoint URL. | | Option 21 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 21 | index | string | Optional name of the Splunk index where logs are written. | | Option 21 | indexed_fields | [string] | List of log field names to send as indexed fields to Splunk HEC. Available only when `encoding` is `json`. | | Option 21 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 21 | sourcetype | string | The Splunk sourcetype to assign to log events. | | Option 21 | token_key | string | Name of the environment variable or secret that holds the Splunk HEC token. | | Option 21 | type [*required*] | enum | The destination type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | destinations | Option 22 | object | The `sumo_logic` destination forwards logs to Sumo Logic. | **Supported pipeline types:** logs | | Option 22 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 22 | encoding | enum | The output encoding format. Allowed enum values: `json,raw_message,logfmt` | | Option 22 | endpoint_url_key | string | Name of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL. | | Option 22 | header_custom_fields | [object] | A list of custom headers to include in the request to Sumo Logic. | | header_custom_fields | name [*required*] | string | The header field name. | | header_custom_fields | value [*required*] | string | The header field value. | | Option 22 | header_host_name | string | Optional override for the host name header. | | Option 22 | header_source_category | string | Optional override for the source category header. | | Option 22 | header_source_name | string | Optional override for the source name header. | | Option 22 | id [*required*] | string | The unique identifier for this component. | | Option 22 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 22 | type [*required*] | enum | The destination type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | destinations | Option 23 | object | The `syslog_ng` destination forwards logs to an external `syslog-ng` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 23 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 23 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog-ng server endpoint URL. | | Option 23 | id [*required*] | string | The unique identifier for this component. | | Option 23 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 23 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 23 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 23 | type [*required*] | enum | The destination type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | destinations | Option 24 | object | The `datadog_metrics` destination forwards metrics to Datadog. | **Supported pipeline types:** metrics | | Option 24 | id [*required*] | string | The unique identifier for this component. | | Option 24 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 24 | type [*required*] | enum | The destination type. The value should always be `datadog_metrics`. Allowed enum values: `datadog_metrics` | | config | pipeline_type | enum | The type of data being ingested. Defaults to `logs` if not specified. Allowed enum values: `logs,metrics` | | config | processor_groups | [object] | A list of processor groups that transform or enrich log data. | | processor_groups | display_name | string | The display name for a component. | | processor_groups | enabled [*required*] | boolean | Whether this processor group is enabled. | | processor_groups | id [*required*] | string | The unique identifier for the processor group. | | processor_groups | include [*required*] | string | Conditional expression for when this processor group should execute. | | processor_groups | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processor_groups | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | processors | [object] | **DEPRECATED**: A list of processor groups that transform or enrich log data. | **Deprecated:** This field is deprecated, you should now use the processor_groups field. | | processors | display_name | string | The display name for a component. | | processors | enabled [*required*] | boolean | Whether this processor group is enabled. | | processors | id [*required*] | string | The unique identifier for the processor group. | | processors | include [*required*] | string | Conditional expression for when this processor group should execute. | | processors | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processors | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | sources [*required*] | [ ] | A list of configured data sources for the pipeline. | | sources | Option 1 | object | The `datadog_agent` source collects logs/metrics from the Datadog Agent. | **Supported pipeline types:** logs, metrics | | Option 1 | address_key | string | Name of the environment variable or secret that holds the listen address for the Datadog Agent source. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | type [*required*] | enum | The source type. The value should always be `datadog_agent`. Allowed enum values: `datadog_agent` | | sources | Option 2 | object | The `amazon_data_firehose` source ingests logs from AWS Data Firehose. | **Supported pipeline types:** logs | | Option 2 | address_key | string | Name of the environment variable or secret that holds the Firehose delivery stream address. | | Option 2 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 2 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 2 | type [*required*] | enum | The source type. The value should always be `amazon_data_firehose`. Allowed enum values: `amazon_data_firehose` | | sources | Option 3 | object | The `amazon_s3` source ingests logs from an Amazon S3 bucket. It supports AWS authentication and TLS encryption. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | region [*required*] | string | AWS region where the S3 bucket resides. | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The source type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | Option 3 | url_key | string | Name of the environment variable or secret that holds the S3 bucket URL. | | sources | Option 4 | object | The `fluent_bit` source ingests logs from Fluent Bit. | **Supported pipeline types:** logs | | Option 4 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent Bit receiver. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 4 | type [*required*] | enum | The source type. The value should always be `fluent_bit`. Allowed enum values: `fluent_bit` | | sources | Option 5 | object | The `fluentd` source ingests logs from a Fluentd-compatible service. | **Supported pipeline types:** logs | | Option 5 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent receiver. | | Option 5 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The source type. The value should always be `fluentd. Allowed enum values: `fluentd` | | sources | Option 6 | object | The `google_pubsub` source ingests logs from a Google Cloud Pub/Sub subscription. | **Supported pipeline types:** logs | | Option 6 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 6 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub subscription. | | Option 6 | subscription [*required*] | string | The Pub/Sub subscription name from which messages are consumed. | | Option 6 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 6 | type [*required*] | enum | The source type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | sources | Option 7 | object | The `http_client` source scrapes logs from HTTP endpoints at regular intervals. | **Supported pipeline types:** logs | | Option 7 | auth_strategy | enum | Optional authentication strategy for HTTP requests. Allowed enum values: `none,basic,bearer,custom` | | Option 7 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 7 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the HTTP endpoint URL to scrape. | | Option 7 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 7 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 7 | scrape_interval_secs | int64 | The interval (in seconds) between HTTP scrape requests. | | Option 7 | scrape_timeout_secs | int64 | The timeout (in seconds) for each scrape request. | | Option 7 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 7 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 7 | type [*required*] | enum | The source type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 7 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | sources | Option 8 | object | The `http_server` source collects logs over HTTP POST from external services. | **Supported pipeline types:** logs | | Option 8 | address_key | string | Name of the environment variable or secret that holds the listen address for the HTTP server. | | Option 8 | auth_strategy [*required*] | enum | HTTP authentication method. Allowed enum values: `none,plain` | | Option 8 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 8 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 8 | id [*required*] | string | Unique ID for the HTTP server source. | | Option 8 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `plain`). | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | type [*required*] | enum | The source type. The value should always be `http_server`. Allowed enum values: `http_server` | | Option 8 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `plain`). | | sources | Option 9 | object | The `kafka` source ingests data from Apache Kafka topics. | **Supported pipeline types:** logs | | Option 9 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 9 | group_id [*required*] | string | Consumer group ID used by the Kafka client. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 9 | librdkafka_options | [object] | Optional list of advanced Kafka client configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 9 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 9 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 9 | topics [*required*] | [string] | A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified. | | Option 9 | type [*required*] | enum | The source type. The value should always be `kafka`. Allowed enum values: `kafka` | | sources | Option 10 | object | The `logstash` source ingests logs from a Logstash forwarder. | **Supported pipeline types:** logs | | Option 10 | address_key | string | Name of the environment variable or secret that holds the listen address for the Logstash receiver. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 10 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 10 | type [*required*] | enum | The source type. The value should always be `logstash`. Allowed enum values: `logstash` | | sources | Option 11 | object | The `rsyslog` source listens for logs over TCP or UDP from an `rsyslog` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 11 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog receiver. | | Option 11 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 11 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 11 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 11 | type [*required*] | enum | The source type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | sources | Option 12 | object | The `socket` source ingests logs over TCP or UDP. | **Supported pipeline types:** logs | | Option 12 | address_key | string | Name of the environment variable or secret that holds the listen address for the socket. | | Option 12 | framing [*required*] | | Framing method configuration for the socket source. | | framing | Option 1 | object | Byte frames which are delimited by a newline character. | | Option 1 | method [*required*] | enum | Byte frames which are delimited by a newline character. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). | | Option 2 | method [*required*] | enum | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). Allowed enum values: `bytes` | | framing | Option 3 | object | Byte frames which are delimited by a chosen character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used to delimit events. | | Option 3 | method [*required*] | enum | Byte frames which are delimited by a chosen character. Allowed enum values: `character_delimited` | | framing | Option 4 | object | Byte frames according to the octet counting format as per RFC6587. | | Option 4 | method [*required*] | enum | Byte frames according to the octet counting format as per RFC6587. Allowed enum values: `octet_counting` | | framing | Option 5 | object | Byte frames which are chunked GELF messages. | | Option 5 | method [*required*] | enum | Byte frames which are chunked GELF messages. Allowed enum values: `chunked_gelf` | | Option 12 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 12 | mode [*required*] | enum | Protocol used to receive logs. Allowed enum values: `tcp,udp` | | Option 12 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 12 | type [*required*] | enum | The source type. The value should always be `socket`. Allowed enum values: `socket` | | sources | Option 13 | object | The `splunk_hec` source implements the Splunk HTTP Event Collector (HEC) API. | **Supported pipeline types:** logs | | Option 13 | address_key | string | Name of the environment variable or secret that holds the listen address for the HEC API. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | type [*required*] | enum | The source type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | sources | Option 14 | object | The `splunk_tcp` source receives logs from a Splunk Universal Forwarder over TCP. TLS is supported for secure transmission. | **Supported pipeline types:** logs | | Option 14 | address_key | string | Name of the environment variable or secret that holds the listen address for the Splunk TCP receiver. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | type [*required*] | enum | The source type. Always `splunk_tcp`. Allowed enum values: `splunk_tcp` | | sources | Option 15 | object | The `sumo_logic` source receives logs from Sumo Logic collectors. | **Supported pipeline types:** logs | | Option 15 | address_key | string | Name of the environment variable or secret that holds the listen address for the Sumo Logic receiver. | | Option 15 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 15 | type [*required*] | enum | The source type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | sources | Option 16 | object | The `syslog_ng` source listens for logs over TCP or UDP from a `syslog-ng` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 16 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog-ng receiver. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 16 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 16 | type [*required*] | enum | The source type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | sources | Option 17 | object | The `opentelemetry` source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP. | **Supported pipeline types:** logs, metrics | | Option 17 | grpc_address_key | string | Environment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | http_address_key | string | Environment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 17 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 17 | type [*required*] | enum | The source type. The value should always be `opentelemetry`. Allowed enum values: `opentelemetry` | | config | use_legacy_search_syntax | boolean | Set to `true` to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to `false`. The legacy syntax is deprecated and will eventually be removed. Requires Observability Pipelines Worker 2.11 or later. See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax/) for more information. | | attributes | name [*required*] | string | Name of the pipeline. | | data | id [*required*] | string | Unique identifier for the pipeline. | | data | type [*required*] | string | The resource type identifier. For pipeline resources, this should always be set to `pipelines`. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": { "destinations": [ { "id": "updated-datadog-logs-destination-id", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Updated Pipeline Name" }, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Top-level schema representing a pipeline. | Parent field | Field | Type | Description | | -------------------- | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Contains the pipeline's ID, type, and configuration attributes. | | data | attributes [*required*] | object | Defines the pipeline's name and its components (sources, processors, and destinations). | | attributes | config [*required*] | object | Specifies the pipeline's configuration, including its sources, processors, and destinations. | | config | destinations [*required*] | [ ] | A list of destination components where processed logs are sent. | | destinations | Option 1 | object | The `http_client` destination sends data to an HTTP endpoint. | **Supported pipeline types:** logs, metrics | | Option 1 | auth_strategy | enum | HTTP authentication strategy. Allowed enum values: `none,basic,bearer` | | Option 1 | compression | object | Compression configuration for HTTP requests. | | compression | algorithm [*required*] | enum | Compression algorithm. Allowed enum values: `gzip` | | Option 1 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 1 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json` | | Option 1 | id [*required*] | string | The unique identifier for this component. | | Option 1 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 1 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 1 | type [*required*] | enum | The destination type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 1 | uri_key | string | Name of the environment variable or secret that holds the HTTP endpoint URI. | | Option 1 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | destinations | Option 2 | object | The `amazon_opensearch` destination writes logs to Amazon OpenSearch. | **Supported pipeline types:** logs | | Option 2 | auth [*required*] | object | Authentication settings for the Amazon OpenSearch destination. The `strategy` field determines whether basic or AWS-based authentication is used. | | auth | assume_role | string | The ARN of the role to assume (used with `aws` strategy). | | auth | aws_region | string | AWS region | | auth | external_id | string | External ID for the assumed role (used with `aws` strategy). | | auth | session_name | string | Session name for the assumed role (used with `aws` strategy). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | Option 2 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 2 | bulk_index | string | The index to write logs to. | | Option 2 | id [*required*] | string | The unique identifier for this component. | | Option 2 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 2 | type [*required*] | enum | The destination type. The value should always be `amazon_opensearch`. Allowed enum values: `amazon_opensearch` | | destinations | Option 3 | object | The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | bucket [*required*] | string | S3 bucket name. | | Option 3 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 3 | id [*required*] | string | Unique identifier for the destination component. | | Option 3 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 3 | key_prefix | string | Optional prefix for object keys. | | Option 3 | region [*required*] | string | AWS region of the S3 bucket. | | Option 3 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The destination type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | destinations | Option 4 | object | The `amazon_s3_generic` destination sends your logs to an Amazon S3 bucket. | **Supported pipeline types:** logs | | Option 4 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 4 | batch_settings | object | Event batching settings | | batch_settings | batch_size | int64 | Maximum batch size in bytes. | | batch_settings | timeout_secs | int64 | Maximum number of seconds to wait before flushing the batch. | | Option 4 | bucket [*required*] | string | S3 bucket name. | | Option 4 | compression [*required*] | | Compression algorithm applied to encoded logs. | | compression | Option 1 | object | Zstd compression. | | Option 1 | algorithm [*required*] | enum | The compression type. Always `zstd`. Allowed enum values: `zstd` | | Option 1 | level [*required*] | int64 | Zstd compression level. | | compression | Option 2 | object | Gzip compression. | | Option 2 | algorithm [*required*] | enum | The compression type. Always `gzip`. Allowed enum values: `gzip` | | Option 2 | level [*required*] | int64 | Gzip compression level. | | compression | Option 3 | object | Snappy compression. | | Option 3 | algorithm [*required*] | enum | The compression type. Always `snappy`. Allowed enum values: `snappy` | | Option 4 | encoding [*required*] | | Encoding format for the destination. | | encoding | Option 1 | object | JSON encoding. | | Option 1 | type [*required*] | enum | The encoding type. Always `json`. Allowed enum values: `json` | | encoding | Option 2 | object | Parquet encoding. | | Option 2 | type [*required*] | enum | The encoding type. Always `parquet`. Allowed enum values: `parquet` | | Option 4 | id [*required*] | string | Unique identifier for the destination component. | | Option 4 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 4 | key_prefix | string | Optional prefix for object keys. | | Option 4 | region [*required*] | string | AWS region of the S3 bucket. | | Option 4 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 4 | type [*required*] | enum | The destination type. Always `amazon_s3_generic`. Allowed enum values: `amazon_s3_generic` | | destinations | Option 5 | object | The `amazon_security_lake` destination sends your logs to Amazon Security Lake. | **Supported pipeline types:** logs | | Option 5 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 5 | bucket [*required*] | string | Name of the Amazon S3 bucket in Security Lake (3-63 characters). | | Option 5 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 5 | custom_source_name [*required*] | string | Custom source name for the logs in Security Lake. | | Option 5 | id [*required*] | string | Unique identifier for the destination component. | | Option 5 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 5 | region [*required*] | string | AWS region of the S3 bucket. | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The destination type. Always `amazon_security_lake`. Allowed enum values: `amazon_security_lake` | | destinations | Option 6 | object | The `azure_storage` destination forwards logs to an Azure Blob Storage container. | **Supported pipeline types:** logs | | Option 6 | blob_prefix | string | Optional prefix for blobs written to the container. | | Option 6 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 6 | connection_string_key | string | Name of the environment variable or secret that holds the Azure Storage connection string. | | Option 6 | container_name [*required*] | string | The name of the Azure Blob Storage container to store logs in. | | Option 6 | id [*required*] | string | The unique identifier for this component. | | Option 6 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 6 | type [*required*] | enum | The destination type. The value should always be `azure_storage`. Allowed enum values: `azure_storage` | | destinations | Option 7 | object | The `cloud_prem` destination sends logs to Datadog CloudPrem. | **Supported pipeline types:** logs | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the CloudPrem endpoint URL. | | Option 7 | id [*required*] | string | The unique identifier for this component. | | Option 7 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 7 | type [*required*] | enum | The destination type. The value should always be `cloud_prem`. Allowed enum values: `cloud_prem` | | destinations | Option 8 | object | The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM. | **Supported pipeline types:** logs | | Option 8 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 8 | compression | object | Compression configuration for log events. | | compression | algorithm [*required*] | enum | Compression algorithm for log events. Allowed enum values: `gzip,zlib` | | compression | level | int64 | Compression level. | | Option 8 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 8 | endpoint_url_key | string | Name of the environment variable or secret that holds the CrowdStrike endpoint URL. | | Option 8 | id [*required*] | string | The unique identifier for this component. | | Option 8 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | token_key | string | Name of the environment variable or secret that holds the CrowdStrike API token. | | Option 8 | type [*required*] | enum | The destination type. The value should always be `crowdstrike_next_gen_siem`. Allowed enum values: `crowdstrike_next_gen_siem` | | destinations | Option 9 | object | The `datadog_logs` destination forwards logs to Datadog Log Management. | **Supported pipeline types:** logs | | Option 9 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 9 | id [*required*] | string | The unique identifier for this component. | | Option 9 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 9 | routes | [object] | A list of routing rules that forward matching logs to Datadog using dedicated API keys. | | routes | api_key_key | string | Name of the environment variable or secret that stores the Datadog API key used by this route. | | routes | include | string | A Datadog search query that determines which logs are forwarded using this route. | | routes | route_id | string | Unique identifier for this route within the destination. | | routes | site | string | Datadog site where matching logs are sent (for example, `us1`). | | Option 9 | type [*required*] | enum | The destination type. The value should always be `datadog_logs`. Allowed enum values: `datadog_logs` | | destinations | Option 10 | object | The `elasticsearch` destination writes logs to an Elasticsearch cluster. | **Supported pipeline types:** logs | | Option 10 | api_version | enum | The Elasticsearch API version to use. Set to `auto` to auto-detect. Allowed enum values: `auto,v6,v7,v8` | | Option 10 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 10 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 10 | bulk_index | string | The index to write logs to in Elasticsearch. | | Option 10 | data_stream | object | Configuration options for writing to Elasticsearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 10 | endpoint_url_key | string | Name of the environment variable or secret that holds the Elasticsearch endpoint URL. | | Option 10 | id [*required*] | string | The unique identifier for this component. | | Option 10 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 10 | type [*required*] | enum | The destination type. The value should always be `elasticsearch`. Allowed enum values: `elasticsearch` | | destinations | Option 11 | object | The `google_chronicle` destination sends logs to Google Chronicle. | **Supported pipeline types:** logs | | Option 11 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 11 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 11 | customer_id [*required*] | string | The Google Chronicle customer ID. | | Option 11 | encoding | enum | The encoding format for the logs sent to Chronicle. Allowed enum values: `json,raw_message` | | Option 11 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Chronicle endpoint URL. | | Option 11 | id [*required*] | string | The unique identifier for this component. | | Option 11 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 11 | log_type | string | The log type metadata associated with the Chronicle destination. | | Option 11 | type [*required*] | enum | The destination type. The value should always be `google_chronicle`. Allowed enum values: `google_chronicle` | | destinations | Option 12 | object | The `google_cloud_storage` destination stores logs in a Google Cloud Storage (GCS) bucket. It requires a bucket name, Google Cloud authentication, and metadata fields. | **Supported pipeline types:** logs | | Option 12 | acl | enum | Access control list setting for objects written to the bucket. Allowed enum values: `private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control` | | Option 12 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 12 | bucket [*required*] | string | Name of the GCS bucket. | | Option 12 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 12 | id [*required*] | string | Unique identifier for the destination component. | | Option 12 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 12 | key_prefix | string | Optional prefix for object keys within the GCS bucket. | | Option 12 | metadata | [object] | Custom metadata to attach to each object uploaded to the GCS bucket. | | metadata | name [*required*] | string | The metadata key. | | metadata | value [*required*] | string | The metadata value. | | Option 12 | storage_class [*required*] | enum | Storage class used for objects stored in GCS. Allowed enum values: `STANDARD,NEARLINE,COLDLINE,ARCHIVE` | | Option 12 | type [*required*] | enum | The destination type. Always `google_cloud_storage`. Allowed enum values: `google_cloud_storage` | | destinations | Option 13 | object | The `google_pubsub` destination publishes logs to a Google Cloud Pub/Sub topic. | **Supported pipeline types:** logs | | Option 13 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 13 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 13 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 13 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL. | | Option 13 | id [*required*] | string | The unique identifier for this component. | | Option 13 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 13 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub topic. | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | topic [*required*] | string | The Pub/Sub topic name to publish logs to. | | Option 13 | type [*required*] | enum | The destination type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | destinations | Option 14 | object | The `kafka` destination sends logs to Apache Kafka topics. | **Supported pipeline types:** logs | | Option 14 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 14 | compression | enum | Compression codec for Kafka messages. Allowed enum values: `none,gzip,snappy,lz4,zstd` | | Option 14 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 14 | headers_key | string | The field name to use for Kafka message headers. | | Option 14 | id [*required*] | string | The unique identifier for this component. | | Option 14 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 14 | key_field | string | The field name to use as the Kafka message key. | | Option 14 | librdkafka_options | [object] | Optional list of advanced Kafka producer configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 14 | message_timeout_ms | int64 | Maximum time in milliseconds to wait for message delivery confirmation. | | Option 14 | rate_limit_duration_secs | int64 | Duration in seconds for the rate limit window. | | Option 14 | rate_limit_num | int64 | Maximum number of messages allowed per rate limit duration. | | Option 14 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 14 | socket_timeout_ms | int64 | Socket timeout in milliseconds for network requests. | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | topic [*required*] | string | The Kafka topic name to publish logs to. | | Option 14 | type [*required*] | enum | The destination type. The value should always be `kafka`. Allowed enum values: `kafka` | | destinations | Option 15 | object | The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel. | **Supported pipeline types:** logs | | Option 15 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 15 | client_id [*required*] | string | Azure AD client ID used for authentication. | | Option 15 | client_secret_key | string | Name of the environment variable or secret that holds the Azure AD client secret. | | Option 15 | dce_uri_key | string | Name of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI. | | Option 15 | dcr_immutable_id [*required*] | string | The immutable ID of the Data Collection Rule (DCR). | | Option 15 | id [*required*] | string | The unique identifier for this component. | | Option 15 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 15 | table [*required*] | string | The name of the Log Analytics table where logs are sent. | | Option 15 | tenant_id [*required*] | string | Azure AD tenant ID. | | Option 15 | type [*required*] | enum | The destination type. The value should always be `microsoft_sentinel`. Allowed enum values: `microsoft_sentinel` | | destinations | Option 16 | object | The `new_relic` destination sends logs to the New Relic platform. | **Supported pipeline types:** logs | | Option 16 | account_id_key | string | Name of the environment variable or secret that holds the New Relic account ID. | | Option 16 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 16 | id [*required*] | string | The unique identifier for this component. | | Option 16 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 16 | license_key_key | string | Name of the environment variable or secret that holds the New Relic license key. | | Option 16 | region [*required*] | enum | The New Relic region. Allowed enum values: `us,eu` | | Option 16 | type [*required*] | enum | The destination type. The value should always be `new_relic`. Allowed enum values: `new_relic` | | destinations | Option 17 | object | The `opensearch` destination writes logs to an OpenSearch cluster. | **Supported pipeline types:** logs | | Option 17 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 17 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 17 | bulk_index | string | The index to write logs to. | | Option 17 | data_stream | object | Configuration options for writing to OpenSearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 17 | endpoint_url_key | string | Name of the environment variable or secret that holds the OpenSearch endpoint URL. | | Option 17 | id [*required*] | string | The unique identifier for this component. | | Option 17 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 17 | type [*required*] | enum | The destination type. The value should always be `opensearch`. Allowed enum values: `opensearch` | | destinations | Option 18 | object | The `rsyslog` destination forwards logs to an external `rsyslog` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 18 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 18 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog server endpoint URL. | | Option 18 | id [*required*] | string | The unique identifier for this component. | | Option 18 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 18 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 18 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 18 | type [*required*] | enum | The destination type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | destinations | Option 19 | object | The `sentinel_one` destination sends logs to SentinelOne. | **Supported pipeline types:** logs | | Option 19 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 19 | id [*required*] | string | The unique identifier for this component. | | Option 19 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 19 | region [*required*] | enum | The SentinelOne region to send logs to. Allowed enum values: `us,eu,ca,data_set_us` | | Option 19 | token_key | string | Name of the environment variable or secret that holds the SentinelOne API token. | | Option 19 | type [*required*] | enum | The destination type. The value should always be `sentinel_one`. Allowed enum values: `sentinel_one` | | destinations | Option 20 | object | The `socket` destination sends logs over TCP or UDP to a remote server. | **Supported pipeline types:** logs | | Option 20 | address_key | string | Name of the environment variable or secret that holds the socket address (host:port). | | Option 20 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 20 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 20 | framing [*required*] | | Framing method configuration. | | framing | Option 1 | object | Each log event is delimited by a newline character. | | Option 1 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod` object. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Event data is not delimited at all. | | Option 2 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod` object. Allowed enum values: `bytes` | | framing | Option 3 | object | Each log event is separated using the specified delimiter character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used as a delimiter. | | Option 3 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod` object. Allowed enum values: `character_delimited` | | Option 20 | id [*required*] | string | The unique identifier for this component. | | Option 20 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 20 | mode [*required*] | enum | Protocol used to send logs. Allowed enum values: `tcp,udp` | | Option 20 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 20 | type [*required*] | enum | The destination type. The value should always be `socket`. Allowed enum values: `socket` | | destinations | Option 21 | object | The `splunk_hec` destination forwards logs to Splunk using the HTTP Event Collector (HEC). | **Supported pipeline types:** logs | | Option 21 | auto_extract_timestamp | boolean | If `true`, Splunk tries to extract timestamps from incoming log events. If `false`, Splunk assigns the time the event was received. | | Option 21 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 21 | encoding | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 21 | endpoint_url_key | string | Name of the environment variable or secret that holds the Splunk HEC endpoint URL. | | Option 21 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 21 | index | string | Optional name of the Splunk index where logs are written. | | Option 21 | indexed_fields | [string] | List of log field names to send as indexed fields to Splunk HEC. Available only when `encoding` is `json`. | | Option 21 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 21 | sourcetype | string | The Splunk sourcetype to assign to log events. | | Option 21 | token_key | string | Name of the environment variable or secret that holds the Splunk HEC token. | | Option 21 | type [*required*] | enum | The destination type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | destinations | Option 22 | object | The `sumo_logic` destination forwards logs to Sumo Logic. | **Supported pipeline types:** logs | | Option 22 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 22 | encoding | enum | The output encoding format. Allowed enum values: `json,raw_message,logfmt` | | Option 22 | endpoint_url_key | string | Name of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL. | | Option 22 | header_custom_fields | [object] | A list of custom headers to include in the request to Sumo Logic. | | header_custom_fields | name [*required*] | string | The header field name. | | header_custom_fields | value [*required*] | string | The header field value. | | Option 22 | header_host_name | string | Optional override for the host name header. | | Option 22 | header_source_category | string | Optional override for the source category header. | | Option 22 | header_source_name | string | Optional override for the source name header. | | Option 22 | id [*required*] | string | The unique identifier for this component. | | Option 22 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 22 | type [*required*] | enum | The destination type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | destinations | Option 23 | object | The `syslog_ng` destination forwards logs to an external `syslog-ng` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 23 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 23 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog-ng server endpoint URL. | | Option 23 | id [*required*] | string | The unique identifier for this component. | | Option 23 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 23 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 23 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 23 | type [*required*] | enum | The destination type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | destinations | Option 24 | object | The `datadog_metrics` destination forwards metrics to Datadog. | **Supported pipeline types:** metrics | | Option 24 | id [*required*] | string | The unique identifier for this component. | | Option 24 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 24 | type [*required*] | enum | The destination type. The value should always be `datadog_metrics`. Allowed enum values: `datadog_metrics` | | config | pipeline_type | enum | The type of data being ingested. Defaults to `logs` if not specified. Allowed enum values: `logs,metrics` | | config | processor_groups | [object] | A list of processor groups that transform or enrich log data. | | processor_groups | display_name | string | The display name for a component. | | processor_groups | enabled [*required*] | boolean | Whether this processor group is enabled. | | processor_groups | id [*required*] | string | The unique identifier for the processor group. | | processor_groups | include [*required*] | string | Conditional expression for when this processor group should execute. | | processor_groups | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processor_groups | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | processors | [object] | **DEPRECATED**: A list of processor groups that transform or enrich log data. | **Deprecated:** This field is deprecated, you should now use the processor_groups field. | | processors | display_name | string | The display name for a component. | | processors | enabled [*required*] | boolean | Whether this processor group is enabled. | | processors | id [*required*] | string | The unique identifier for the processor group. | | processors | include [*required*] | string | Conditional expression for when this processor group should execute. | | processors | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processors | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | sources [*required*] | [ ] | A list of configured data sources for the pipeline. | | sources | Option 1 | object | The `datadog_agent` source collects logs/metrics from the Datadog Agent. | **Supported pipeline types:** logs, metrics | | Option 1 | address_key | string | Name of the environment variable or secret that holds the listen address for the Datadog Agent source. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | type [*required*] | enum | The source type. The value should always be `datadog_agent`. Allowed enum values: `datadog_agent` | | sources | Option 2 | object | The `amazon_data_firehose` source ingests logs from AWS Data Firehose. | **Supported pipeline types:** logs | | Option 2 | address_key | string | Name of the environment variable or secret that holds the Firehose delivery stream address. | | Option 2 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 2 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 2 | type [*required*] | enum | The source type. The value should always be `amazon_data_firehose`. Allowed enum values: `amazon_data_firehose` | | sources | Option 3 | object | The `amazon_s3` source ingests logs from an Amazon S3 bucket. It supports AWS authentication and TLS encryption. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | region [*required*] | string | AWS region where the S3 bucket resides. | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The source type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | Option 3 | url_key | string | Name of the environment variable or secret that holds the S3 bucket URL. | | sources | Option 4 | object | The `fluent_bit` source ingests logs from Fluent Bit. | **Supported pipeline types:** logs | | Option 4 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent Bit receiver. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 4 | type [*required*] | enum | The source type. The value should always be `fluent_bit`. Allowed enum values: `fluent_bit` | | sources | Option 5 | object | The `fluentd` source ingests logs from a Fluentd-compatible service. | **Supported pipeline types:** logs | | Option 5 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent receiver. | | Option 5 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The source type. The value should always be `fluentd. Allowed enum values: `fluentd` | | sources | Option 6 | object | The `google_pubsub` source ingests logs from a Google Cloud Pub/Sub subscription. | **Supported pipeline types:** logs | | Option 6 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 6 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub subscription. | | Option 6 | subscription [*required*] | string | The Pub/Sub subscription name from which messages are consumed. | | Option 6 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 6 | type [*required*] | enum | The source type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | sources | Option 7 | object | The `http_client` source scrapes logs from HTTP endpoints at regular intervals. | **Supported pipeline types:** logs | | Option 7 | auth_strategy | enum | Optional authentication strategy for HTTP requests. Allowed enum values: `none,basic,bearer,custom` | | Option 7 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 7 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the HTTP endpoint URL to scrape. | | Option 7 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 7 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 7 | scrape_interval_secs | int64 | The interval (in seconds) between HTTP scrape requests. | | Option 7 | scrape_timeout_secs | int64 | The timeout (in seconds) for each scrape request. | | Option 7 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 7 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 7 | type [*required*] | enum | The source type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 7 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | sources | Option 8 | object | The `http_server` source collects logs over HTTP POST from external services. | **Supported pipeline types:** logs | | Option 8 | address_key | string | Name of the environment variable or secret that holds the listen address for the HTTP server. | | Option 8 | auth_strategy [*required*] | enum | HTTP authentication method. Allowed enum values: `none,plain` | | Option 8 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 8 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 8 | id [*required*] | string | Unique ID for the HTTP server source. | | Option 8 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `plain`). | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | type [*required*] | enum | The source type. The value should always be `http_server`. Allowed enum values: `http_server` | | Option 8 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `plain`). | | sources | Option 9 | object | The `kafka` source ingests data from Apache Kafka topics. | **Supported pipeline types:** logs | | Option 9 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 9 | group_id [*required*] | string | Consumer group ID used by the Kafka client. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 9 | librdkafka_options | [object] | Optional list of advanced Kafka client configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 9 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 9 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 9 | topics [*required*] | [string] | A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified. | | Option 9 | type [*required*] | enum | The source type. The value should always be `kafka`. Allowed enum values: `kafka` | | sources | Option 10 | object | The `logstash` source ingests logs from a Logstash forwarder. | **Supported pipeline types:** logs | | Option 10 | address_key | string | Name of the environment variable or secret that holds the listen address for the Logstash receiver. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 10 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 10 | type [*required*] | enum | The source type. The value should always be `logstash`. Allowed enum values: `logstash` | | sources | Option 11 | object | The `rsyslog` source listens for logs over TCP or UDP from an `rsyslog` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 11 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog receiver. | | Option 11 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 11 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 11 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 11 | type [*required*] | enum | The source type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | sources | Option 12 | object | The `socket` source ingests logs over TCP or UDP. | **Supported pipeline types:** logs | | Option 12 | address_key | string | Name of the environment variable or secret that holds the listen address for the socket. | | Option 12 | framing [*required*] | | Framing method configuration for the socket source. | | framing | Option 1 | object | Byte frames which are delimited by a newline character. | | Option 1 | method [*required*] | enum | Byte frames which are delimited by a newline character. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). | | Option 2 | method [*required*] | enum | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). Allowed enum values: `bytes` | | framing | Option 3 | object | Byte frames which are delimited by a chosen character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used to delimit events. | | Option 3 | method [*required*] | enum | Byte frames which are delimited by a chosen character. Allowed enum values: `character_delimited` | | framing | Option 4 | object | Byte frames according to the octet counting format as per RFC6587. | | Option 4 | method [*required*] | enum | Byte frames according to the octet counting format as per RFC6587. Allowed enum values: `octet_counting` | | framing | Option 5 | object | Byte frames which are chunked GELF messages. | | Option 5 | method [*required*] | enum | Byte frames which are chunked GELF messages. Allowed enum values: `chunked_gelf` | | Option 12 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 12 | mode [*required*] | enum | Protocol used to receive logs. Allowed enum values: `tcp,udp` | | Option 12 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 12 | type [*required*] | enum | The source type. The value should always be `socket`. Allowed enum values: `socket` | | sources | Option 13 | object | The `splunk_hec` source implements the Splunk HTTP Event Collector (HEC) API. | **Supported pipeline types:** logs | | Option 13 | address_key | string | Name of the environment variable or secret that holds the listen address for the HEC API. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | type [*required*] | enum | The source type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | sources | Option 14 | object | The `splunk_tcp` source receives logs from a Splunk Universal Forwarder over TCP. TLS is supported for secure transmission. | **Supported pipeline types:** logs | | Option 14 | address_key | string | Name of the environment variable or secret that holds the listen address for the Splunk TCP receiver. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | type [*required*] | enum | The source type. Always `splunk_tcp`. Allowed enum values: `splunk_tcp` | | sources | Option 15 | object | The `sumo_logic` source receives logs from Sumo Logic collectors. | **Supported pipeline types:** logs | | Option 15 | address_key | string | Name of the environment variable or secret that holds the listen address for the Sumo Logic receiver. | | Option 15 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 15 | type [*required*] | enum | The source type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | sources | Option 16 | object | The `syslog_ng` source listens for logs over TCP or UDP from a `syslog-ng` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 16 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog-ng receiver. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 16 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 16 | type [*required*] | enum | The source type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | sources | Option 17 | object | The `opentelemetry` source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP. | **Supported pipeline types:** logs, metrics | | Option 17 | grpc_address_key | string | Environment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | http_address_key | string | Environment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 17 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 17 | type [*required*] | enum | The source type. The value should always be `opentelemetry`. Allowed enum values: `opentelemetry` | | config | use_legacy_search_syntax | boolean | Set to `true` to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to `false`. The legacy syntax is deprecated and will eventually be removed. Requires Observability Pipelines Worker 2.11 or later. See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax/) for more information. | | attributes | name [*required*] | string | Name of the pipeline. | | data | id [*required*] | string | Unique identifier for the pipeline. | | data | type [*required*] | string | The resource type identifier. For pipeline resources, this should always be set to `pipelines`. | {% /tab %} {% tab title="Example" %} ```json { "data": { "attributes": { "config": { "destinations": [ { "auth_strategy": "basic", "compression": { "algorithm": "gzip" }, "custom_key": "HTTP_AUTH_CUSTOM_HEADER", "encoding": "json", "id": "http-client-destination", "inputs": [ "filter-processor" ], "password_key": "HTTP_AUTH_PASSWORD", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "token_key": "HTTP_AUTH_TOKEN", "type": "http_client", "uri_key": "HTTP_DESTINATION_URI", "username_key": "HTTP_AUTH_USERNAME" } ], "pipeline_type": "logs", "processor_groups": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "processors": [ { "display_name": "my component", "enabled": true, "id": "grouped-processors", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ [] ] } ], "sources": [ { "address_key": "DATADOG_AGENT_ADDRESS", "id": "datadog-agent-source", "tls": { "ca_file": "string", "crt_file": "/path/to/cert.crt", "key_file": "string", "key_pass_key": "TLS_KEY_PASSPHRASE" }, "type": "datadog_agent" } ], "use_legacy_search_syntax": false }, "name": "Main Observability Pipeline" }, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines" } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Not Authorized {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="409" %} Conflict {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport pipeline_id="CHANGE_ME"\# Curl commandcurl -X PUT "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/${pipeline_id}" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "config": { "destinations": [ { "id": "updated-datadog-logs-destination-id", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Updated Pipeline Name" }, "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "type": "pipelines" } } EOF ##### ```go // Update a pipeline returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { // there is a valid "pipeline" in the system PipelineDataID := os.Getenv("PIPELINE_DATA_ID") body := datadogV2.ObservabilityPipeline{ Data: datadogV2.ObservabilityPipelineData{ Attributes: datadogV2.ObservabilityPipelineDataAttributes{ Config: datadogV2.ObservabilityPipelineConfig{ Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{ datadogV2.ObservabilityPipelineConfigDestinationItem{ ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{ Id: "updated-datadog-logs-destination-id", Inputs: []string{ "my-processor-group", }, Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS, }}, }, ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{ { Enabled: true, Id: "my-processor-group", Include: "service:my-service", Inputs: []string{ "datadog-agent-source", }, Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{ datadogV2.ObservabilityPipelineConfigProcessorItem{ ObservabilityPipelineFilterProcessor: &datadogV2.ObservabilityPipelineFilterProcessor{ Enabled: true, Id: "filter-processor", Include: "status:error", Type: datadogV2.OBSERVABILITYPIPELINEFILTERPROCESSORTYPE_FILTER, }}, }, }, }, Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{ datadogV2.ObservabilityPipelineConfigSourceItem{ ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{ Id: "datadog-agent-source", Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT, }}, }, }, Name: "Updated Pipeline Name", }, Id: PipelineDataID, Type: "pipelines", }, } ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) resp, r, err := api.UpdatePipeline(ctx, PipelineDataID, body) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.UpdatePipeline`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.UpdatePipeline`:\n%s\n", responseContent) } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=go) and then save the example to `main.go` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" go run "main.go" ##### ```java // Update a pipeline returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; import com.datadog.api.client.v2.model.ObservabilityPipeline; import com.datadog.api.client.v2.model.ObservabilityPipelineConfig; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem; import com.datadog.api.client.v2.model.ObservabilityPipelineData; import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType; import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessor; import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessorType; import java.util.Collections; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); // there is a valid "pipeline" in the system String PIPELINE_DATA_ID = System.getenv("PIPELINE_DATA_ID"); ObservabilityPipeline body = new ObservabilityPipeline() .data( new ObservabilityPipelineData() .attributes( new ObservabilityPipelineDataAttributes() .config( new ObservabilityPipelineConfig() .destinations( Collections.singletonList( new ObservabilityPipelineConfigDestinationItem( new ObservabilityPipelineDatadogLogsDestination() .id("updated-datadog-logs-destination-id") .inputs( Collections.singletonList( "my-processor-group")) .type( ObservabilityPipelineDatadogLogsDestinationType .DATADOG_LOGS)))) .processorGroups( Collections.singletonList( new ObservabilityPipelineConfigProcessorGroup() .enabled(true) .id("my-processor-group") .include("service:my-service") .inputs( Collections.singletonList( "datadog-agent-source")) .processors( Collections.singletonList( new ObservabilityPipelineConfigProcessorItem( new ObservabilityPipelineFilterProcessor() .enabled(true) .id("filter-processor") .include("status:error") .type( ObservabilityPipelineFilterProcessorType .FILTER)))))) .sources( Collections.singletonList( new ObservabilityPipelineConfigSourceItem( new ObservabilityPipelineDatadogAgentSource() .id("datadog-agent-source") .type( ObservabilityPipelineDatadogAgentSourceType .DATADOG_AGENT))))) .name("Updated Pipeline Name")) .id(PIPELINE_DATA_ID) .type("pipelines")); try { ObservabilityPipeline result = apiInstance.updatePipeline(PIPELINE_DATA_ID, body); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#updatePipeline"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=java) and then save the example to `Example.java` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" java "Example.java" ##### ```python """ Update a pipeline returns "OK" response """ from os import environ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi from datadog_api_client.v2.model.observability_pipeline import ObservabilityPipeline from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig from datadog_api_client.v2.model.observability_pipeline_config_processor_group import ( ObservabilityPipelineConfigProcessorGroup, ) from datadog_api_client.v2.model.observability_pipeline_data import ObservabilityPipelineData from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import ( ObservabilityPipelineDatadogAgentSource, ) from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import ( ObservabilityPipelineDatadogAgentSourceType, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import ( ObservabilityPipelineDatadogLogsDestination, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import ( ObservabilityPipelineDatadogLogsDestinationType, ) from datadog_api_client.v2.model.observability_pipeline_filter_processor import ObservabilityPipelineFilterProcessor from datadog_api_client.v2.model.observability_pipeline_filter_processor_type import ( ObservabilityPipelineFilterProcessorType, ) # there is a valid "pipeline" in the system PIPELINE_DATA_ID = environ["PIPELINE_DATA_ID"] body = ObservabilityPipeline( data=ObservabilityPipelineData( attributes=ObservabilityPipelineDataAttributes( config=ObservabilityPipelineConfig( destinations=[ ObservabilityPipelineDatadogLogsDestination( id="updated-datadog-logs-destination-id", inputs=[ "my-processor-group", ], type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS, ), ], processor_groups=[ ObservabilityPipelineConfigProcessorGroup( enabled=True, id="my-processor-group", include="service:my-service", inputs=[ "datadog-agent-source", ], processors=[ ObservabilityPipelineFilterProcessor( enabled=True, id="filter-processor", include="status:error", type=ObservabilityPipelineFilterProcessorType.FILTER, ), ], ), ], sources=[ ObservabilityPipelineDatadogAgentSource( id="datadog-agent-source", type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT, ), ], ), name="Updated Pipeline Name", ), id=PIPELINE_DATA_ID, type="pipelines", ), ) configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) response = api_instance.update_pipeline(pipeline_id=PIPELINE_DATA_ID, body=body) print(response) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=python) and then save the example to `example.py` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" python3 "example.py" ##### ```ruby # Update a pipeline returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new # there is a valid "pipeline" in the system PIPELINE_DATA_ID = ENV["PIPELINE_DATA_ID"] body = DatadogAPIClient::V2::ObservabilityPipeline.new({ data: DatadogAPIClient::V2::ObservabilityPipelineData.new({ attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({ config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({ destinations: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({ id: "updated-datadog-logs-destination-id", inputs: [ "my-processor-group", ], type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, }), ], processor_groups: [ DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({ enabled: true, id: "my-processor-group", include: "service:my-service", inputs: [ "datadog-agent-source", ], processors: [ DatadogAPIClient::V2::ObservabilityPipelineFilterProcessor.new({ enabled: true, id: "filter-processor", include: "status:error", type: DatadogAPIClient::V2::ObservabilityPipelineFilterProcessorType::FILTER, }), ], }), ], sources: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({ id: "datadog-agent-source", type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, }), ], }), name: "Updated Pipeline Name", }), id: PIPELINE_DATA_ID, type: "pipelines", }), }) p api_instance.update_pipeline(PIPELINE_DATA_ID, body) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=ruby) and then save the example to `example.rb` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" rb "example.rb" ##### ```rust // Update a pipeline returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; use datadog_api_client::datadogV2::model::ObservabilityPipeline; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineData; use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType; use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessor; use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessorType; #[tokio::main] async fn main() { // there is a valid "pipeline" in the system let pipeline_data_id = std::env::var("PIPELINE_DATA_ID").unwrap(); let body = ObservabilityPipeline::new( ObservabilityPipelineData::new( ObservabilityPipelineDataAttributes::new( ObservabilityPipelineConfig::new( vec![ ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination( Box::new( ObservabilityPipelineDatadogLogsDestination::new( "updated-datadog-logs-destination-id".to_string(), vec!["my-processor-group".to_string()], ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, ), ), ) ], vec![ ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource( Box::new( ObservabilityPipelineDatadogAgentSource::new( "datadog-agent-source".to_string(), ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, ), ), ) ], ).processor_groups( vec![ ObservabilityPipelineConfigProcessorGroup::new( true, "my-processor-group".to_string(), "service:my-service".to_string(), vec!["datadog-agent-source".to_string()], vec![ ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineFilterProcessor( Box::new( ObservabilityPipelineFilterProcessor::new( true, "filter-processor".to_string(), "status:error".to_string(), ObservabilityPipelineFilterProcessorType::FILTER, ), ), ) ], ) ], ), "Updated Pipeline Name".to_string(), ), pipeline_data_id.clone(), "pipelines".to_string(), ), ); let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api.update_pipeline(pipeline_data_id.clone(), body).await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=rust) and then save the example to `src/main.rs` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" cargo run ##### ```typescript /** * Update a pipeline returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); // there is a valid "pipeline" in the system const PIPELINE_DATA_ID = process.env.PIPELINE_DATA_ID as string; const params: v2.ObservabilityPipelinesApiUpdatePipelineRequest = { body: { data: { attributes: { config: { destinations: [ { id: "updated-datadog-logs-destination-id", inputs: ["my-processor-group"], type: "datadog_logs", }, ], processorGroups: [ { enabled: true, id: "my-processor-group", include: "service:my-service", inputs: ["datadog-agent-source"], processors: [ { enabled: true, id: "filter-processor", include: "status:error", type: "filter", }, ], }, ], sources: [ { id: "datadog-agent-source", type: "datadog_agent", }, ], }, name: "Updated Pipeline Name", }, id: PIPELINE_DATA_ID, type: "pipelines", }, }, pipelineId: PIPELINE_DATA_ID, }; apiInstance .updatePipeline(params) .then((data: v2.ObservabilityPipeline) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=typescript) and then save the example to `example.ts` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" tsc "example.ts" {% /tab %} ## Delete a pipeline{% #delete-a-pipeline %} {% tab title="v2" %} | Datadog site | API endpoint | | ----------------- | --------------------------------------------------------------------------------- | | ap1.datadoghq.com | DELETE https://api.ap1.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | ap2.datadoghq.com | DELETE https://api.ap2.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.datadoghq.eu | DELETE https://api.datadoghq.eu/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.ddog-gov.com | DELETE https://api.ddog-gov.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | app.datadoghq.com | DELETE https://api.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | us3.datadoghq.com | DELETE https://api.us3.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | | us5.datadoghq.com | DELETE https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/{pipeline_id} | ### Overview Delete a pipeline. This endpoint requires the `observability_pipelines_delete` permission. ### Arguments #### Path Parameters | Name | Type | Description | | ----------------------------- | ------ | --------------------------------- | | pipeline_id [*required*] | string | The ID of the pipeline to delete. | ### Response {% tab title="204" %} OK {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="404" %} Not Found {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="409" %} Conflict {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Path parametersexport pipeline_id="CHANGE_ME"\# Curl commandcurl -X DELETE "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/${pipeline_id}" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" ##### ```python """ Delete a pipeline returns "OK" response """ from os import environ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi # there is a valid "pipeline" in the system PIPELINE_DATA_ID = environ["PIPELINE_DATA_ID"] configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) api_instance.delete_pipeline( pipeline_id=PIPELINE_DATA_ID, ) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=python) and then save the example to `example.py` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" python3 "example.py" ##### ```ruby # Delete a pipeline returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new # there is a valid "pipeline" in the system PIPELINE_DATA_ID = ENV["PIPELINE_DATA_ID"] api_instance.delete_pipeline(PIPELINE_DATA_ID) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=ruby) and then save the example to `example.rb` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" rb "example.rb" ##### ```go // Delete a pipeline returns "OK" response package main import ( "context" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { // there is a valid "pipeline" in the system PipelineDataID := os.Getenv("PIPELINE_DATA_ID") ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) r, err := api.DeletePipeline(ctx, PipelineDataID) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.DeletePipeline`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=go) and then save the example to `main.go` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" go run "main.go" ##### ```java // Delete a pipeline returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); // there is a valid "pipeline" in the system String PIPELINE_DATA_ID = System.getenv("PIPELINE_DATA_ID"); try { apiInstance.deletePipeline(PIPELINE_DATA_ID); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#deletePipeline"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=java) and then save the example to `Example.java` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" java "Example.java" ##### ```rust // Delete a pipeline returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; #[tokio::main] async fn main() { // there is a valid "pipeline" in the system let pipeline_data_id = std::env::var("PIPELINE_DATA_ID").unwrap(); let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api.delete_pipeline(pipeline_data_id.clone()).await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=rust) and then save the example to `src/main.rs` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" cargo run ##### ```typescript /** * Delete a pipeline returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); // there is a valid "pipeline" in the system const PIPELINE_DATA_ID = process.env.PIPELINE_DATA_ID as string; const params: v2.ObservabilityPipelinesApiDeletePipelineRequest = { pipelineId: PIPELINE_DATA_ID, }; apiInstance .deletePipeline(params) .then((data: any) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=typescript) and then save the example to `example.ts` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" tsc "example.ts" {% /tab %} ## Validate an observability pipeline{% #validate-an-observability-pipeline %} {% tab title="v2" %} | Datadog site | API endpoint | | ----------------- | -------------------------------------------------------------------------- | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/obs-pipelines/pipelines/validate | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/obs-pipelines/pipelines/validate | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/obs-pipelines/pipelines/validate | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/obs-pipelines/pipelines/validate | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/obs-pipelines/pipelines/validate | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/obs-pipelines/pipelines/validate | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/validate | ### Overview Validates a pipeline configuration without creating or updating any resources. Returns a list of validation errors, if any. This endpoint requires the `observability_pipelines_read` permission. ### Request #### Body Data (required) {% tab title="Model" %} | Parent field | Field | Type | Description | | -------------------- | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | data [*required*] | object | Contains the the pipeline configuration. | | data | attributes [*required*] | object | Defines the pipeline's name and its components (sources, processors, and destinations). | | attributes | config [*required*] | object | Specifies the pipeline's configuration, including its sources, processors, and destinations. | | config | destinations [*required*] | [ ] | A list of destination components where processed logs are sent. | | destinations | Option 1 | object | The `http_client` destination sends data to an HTTP endpoint. | **Supported pipeline types:** logs, metrics | | Option 1 | auth_strategy | enum | HTTP authentication strategy. Allowed enum values: `none,basic,bearer` | | Option 1 | compression | object | Compression configuration for HTTP requests. | | compression | algorithm [*required*] | enum | Compression algorithm. Allowed enum values: `gzip` | | Option 1 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 1 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json` | | Option 1 | id [*required*] | string | The unique identifier for this component. | | Option 1 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 1 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 1 | type [*required*] | enum | The destination type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 1 | uri_key | string | Name of the environment variable or secret that holds the HTTP endpoint URI. | | Option 1 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | destinations | Option 2 | object | The `amazon_opensearch` destination writes logs to Amazon OpenSearch. | **Supported pipeline types:** logs | | Option 2 | auth [*required*] | object | Authentication settings for the Amazon OpenSearch destination. The `strategy` field determines whether basic or AWS-based authentication is used. | | auth | assume_role | string | The ARN of the role to assume (used with `aws` strategy). | | auth | aws_region | string | AWS region | | auth | external_id | string | External ID for the assumed role (used with `aws` strategy). | | auth | session_name | string | Session name for the assumed role (used with `aws` strategy). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | Option 2 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 2 | bulk_index | string | The index to write logs to. | | Option 2 | id [*required*] | string | The unique identifier for this component. | | Option 2 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 2 | type [*required*] | enum | The destination type. The value should always be `amazon_opensearch`. Allowed enum values: `amazon_opensearch` | | destinations | Option 3 | object | The `amazon_s3` destination sends your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | bucket [*required*] | string | S3 bucket name. | | Option 3 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 3 | id [*required*] | string | Unique identifier for the destination component. | | Option 3 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 3 | key_prefix | string | Optional prefix for object keys. | | Option 3 | region [*required*] | string | AWS region of the S3 bucket. | | Option 3 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The destination type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | destinations | Option 4 | object | The `amazon_s3_generic` destination sends your logs to an Amazon S3 bucket. | **Supported pipeline types:** logs | | Option 4 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 4 | batch_settings | object | Event batching settings | | batch_settings | batch_size | int64 | Maximum batch size in bytes. | | batch_settings | timeout_secs | int64 | Maximum number of seconds to wait before flushing the batch. | | Option 4 | bucket [*required*] | string | S3 bucket name. | | Option 4 | compression [*required*] | | Compression algorithm applied to encoded logs. | | compression | Option 1 | object | Zstd compression. | | Option 1 | algorithm [*required*] | enum | The compression type. Always `zstd`. Allowed enum values: `zstd` | | Option 1 | level [*required*] | int64 | Zstd compression level. | | compression | Option 2 | object | Gzip compression. | | Option 2 | algorithm [*required*] | enum | The compression type. Always `gzip`. Allowed enum values: `gzip` | | Option 2 | level [*required*] | int64 | Gzip compression level. | | compression | Option 3 | object | Snappy compression. | | Option 3 | algorithm [*required*] | enum | The compression type. Always `snappy`. Allowed enum values: `snappy` | | Option 4 | encoding [*required*] | | Encoding format for the destination. | | encoding | Option 1 | object | JSON encoding. | | Option 1 | type [*required*] | enum | The encoding type. Always `json`. Allowed enum values: `json` | | encoding | Option 2 | object | Parquet encoding. | | Option 2 | type [*required*] | enum | The encoding type. Always `parquet`. Allowed enum values: `parquet` | | Option 4 | id [*required*] | string | Unique identifier for the destination component. | | Option 4 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 4 | key_prefix | string | Optional prefix for object keys. | | Option 4 | region [*required*] | string | AWS region of the S3 bucket. | | Option 4 | storage_class [*required*] | enum | S3 storage class. Allowed enum values: `STANDARD,REDUCED_REDUNDANCY,INTELLIGENT_TIERING,STANDARD_IA,EXPRESS_ONEZONE,ONEZONE_IA,GLACIER,GLACIER_IR,DEEP_ARCHIVE` | | Option 4 | type [*required*] | enum | The destination type. Always `amazon_s3_generic`. Allowed enum values: `amazon_s3_generic` | | destinations | Option 5 | object | The `amazon_security_lake` destination sends your logs to Amazon Security Lake. | **Supported pipeline types:** logs | | Option 5 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 5 | bucket [*required*] | string | Name of the Amazon S3 bucket in Security Lake (3-63 characters). | | Option 5 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 5 | custom_source_name [*required*] | string | Custom source name for the logs in Security Lake. | | Option 5 | id [*required*] | string | Unique identifier for the destination component. | | Option 5 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 5 | region [*required*] | string | AWS region of the S3 bucket. | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The destination type. Always `amazon_security_lake`. Allowed enum values: `amazon_security_lake` | | destinations | Option 6 | object | The `azure_storage` destination forwards logs to an Azure Blob Storage container. | **Supported pipeline types:** logs | | Option 6 | blob_prefix | string | Optional prefix for blobs written to the container. | | Option 6 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 6 | connection_string_key | string | Name of the environment variable or secret that holds the Azure Storage connection string. | | Option 6 | container_name [*required*] | string | The name of the Azure Blob Storage container to store logs in. | | Option 6 | id [*required*] | string | The unique identifier for this component. | | Option 6 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 6 | type [*required*] | enum | The destination type. The value should always be `azure_storage`. Allowed enum values: `azure_storage` | | destinations | Option 7 | object | The `cloud_prem` destination sends logs to Datadog CloudPrem. | **Supported pipeline types:** logs | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the CloudPrem endpoint URL. | | Option 7 | id [*required*] | string | The unique identifier for this component. | | Option 7 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 7 | type [*required*] | enum | The destination type. The value should always be `cloud_prem`. Allowed enum values: `cloud_prem` | | destinations | Option 8 | object | The `crowdstrike_next_gen_siem` destination forwards logs to CrowdStrike Next Gen SIEM. | **Supported pipeline types:** logs | | Option 8 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 8 | compression | object | Compression configuration for log events. | | compression | algorithm [*required*] | enum | Compression algorithm for log events. Allowed enum values: `gzip,zlib` | | compression | level | int64 | Compression level. | | Option 8 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 8 | endpoint_url_key | string | Name of the environment variable or secret that holds the CrowdStrike endpoint URL. | | Option 8 | id [*required*] | string | The unique identifier for this component. | | Option 8 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | token_key | string | Name of the environment variable or secret that holds the CrowdStrike API token. | | Option 8 | type [*required*] | enum | The destination type. The value should always be `crowdstrike_next_gen_siem`. Allowed enum values: `crowdstrike_next_gen_siem` | | destinations | Option 9 | object | The `datadog_logs` destination forwards logs to Datadog Log Management. | **Supported pipeline types:** logs | | Option 9 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 9 | id [*required*] | string | The unique identifier for this component. | | Option 9 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 9 | routes | [object] | A list of routing rules that forward matching logs to Datadog using dedicated API keys. | | routes | api_key_key | string | Name of the environment variable or secret that stores the Datadog API key used by this route. | | routes | include | string | A Datadog search query that determines which logs are forwarded using this route. | | routes | route_id | string | Unique identifier for this route within the destination. | | routes | site | string | Datadog site where matching logs are sent (for example, `us1`). | | Option 9 | type [*required*] | enum | The destination type. The value should always be `datadog_logs`. Allowed enum values: `datadog_logs` | | destinations | Option 10 | object | The `elasticsearch` destination writes logs to an Elasticsearch cluster. | **Supported pipeline types:** logs | | Option 10 | api_version | enum | The Elasticsearch API version to use. Set to `auto` to auto-detect. Allowed enum values: `auto,v6,v7,v8` | | Option 10 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 10 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 10 | bulk_index | string | The index to write logs to in Elasticsearch. | | Option 10 | data_stream | object | Configuration options for writing to Elasticsearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 10 | endpoint_url_key | string | Name of the environment variable or secret that holds the Elasticsearch endpoint URL. | | Option 10 | id [*required*] | string | The unique identifier for this component. | | Option 10 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 10 | type [*required*] | enum | The destination type. The value should always be `elasticsearch`. Allowed enum values: `elasticsearch` | | destinations | Option 11 | object | The `google_chronicle` destination sends logs to Google Chronicle. | **Supported pipeline types:** logs | | Option 11 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 11 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 11 | customer_id [*required*] | string | The Google Chronicle customer ID. | | Option 11 | encoding | enum | The encoding format for the logs sent to Chronicle. Allowed enum values: `json,raw_message` | | Option 11 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Chronicle endpoint URL. | | Option 11 | id [*required*] | string | The unique identifier for this component. | | Option 11 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 11 | log_type | string | The log type metadata associated with the Chronicle destination. | | Option 11 | type [*required*] | enum | The destination type. The value should always be `google_chronicle`. Allowed enum values: `google_chronicle` | | destinations | Option 12 | object | The `google_cloud_storage` destination stores logs in a Google Cloud Storage (GCS) bucket. It requires a bucket name, Google Cloud authentication, and metadata fields. | **Supported pipeline types:** logs | | Option 12 | acl | enum | Access control list setting for objects written to the bucket. Allowed enum values: `private,project-private,public-read,authenticated-read,bucket-owner-read,bucket-owner-full-control` | | Option 12 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 12 | bucket [*required*] | string | Name of the GCS bucket. | | Option 12 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 12 | id [*required*] | string | Unique identifier for the destination component. | | Option 12 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 12 | key_prefix | string | Optional prefix for object keys within the GCS bucket. | | Option 12 | metadata | [object] | Custom metadata to attach to each object uploaded to the GCS bucket. | | metadata | name [*required*] | string | The metadata key. | | metadata | value [*required*] | string | The metadata value. | | Option 12 | storage_class [*required*] | enum | Storage class used for objects stored in GCS. Allowed enum values: `STANDARD,NEARLINE,COLDLINE,ARCHIVE` | | Option 12 | type [*required*] | enum | The destination type. Always `google_cloud_storage`. Allowed enum values: `google_cloud_storage` | | destinations | Option 13 | object | The `google_pubsub` destination publishes logs to a Google Cloud Pub/Sub topic. | **Supported pipeline types:** logs | | Option 13 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 13 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 13 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 13 | endpoint_url_key | string | Name of the environment variable or secret that holds the Google Cloud Pub/Sub endpoint URL. | | Option 13 | id [*required*] | string | The unique identifier for this component. | | Option 13 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 13 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub topic. | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | topic [*required*] | string | The Pub/Sub topic name to publish logs to. | | Option 13 | type [*required*] | enum | The destination type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | destinations | Option 14 | object | The `kafka` destination sends logs to Apache Kafka topics. | **Supported pipeline types:** logs | | Option 14 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 14 | compression | enum | Compression codec for Kafka messages. Allowed enum values: `none,gzip,snappy,lz4,zstd` | | Option 14 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 14 | headers_key | string | The field name to use for Kafka message headers. | | Option 14 | id [*required*] | string | The unique identifier for this component. | | Option 14 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 14 | key_field | string | The field name to use as the Kafka message key. | | Option 14 | librdkafka_options | [object] | Optional list of advanced Kafka producer configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 14 | message_timeout_ms | int64 | Maximum time in milliseconds to wait for message delivery confirmation. | | Option 14 | rate_limit_duration_secs | int64 | Duration in seconds for the rate limit window. | | Option 14 | rate_limit_num | int64 | Maximum number of messages allowed per rate limit duration. | | Option 14 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 14 | socket_timeout_ms | int64 | Socket timeout in milliseconds for network requests. | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | topic [*required*] | string | The Kafka topic name to publish logs to. | | Option 14 | type [*required*] | enum | The destination type. The value should always be `kafka`. Allowed enum values: `kafka` | | destinations | Option 15 | object | The `microsoft_sentinel` destination forwards logs to Microsoft Sentinel. | **Supported pipeline types:** logs | | Option 15 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 15 | client_id [*required*] | string | Azure AD client ID used for authentication. | | Option 15 | client_secret_key | string | Name of the environment variable or secret that holds the Azure AD client secret. | | Option 15 | dce_uri_key | string | Name of the environment variable or secret that holds the Data Collection Endpoint (DCE) URI. | | Option 15 | dcr_immutable_id [*required*] | string | The immutable ID of the Data Collection Rule (DCR). | | Option 15 | id [*required*] | string | The unique identifier for this component. | | Option 15 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 15 | table [*required*] | string | The name of the Log Analytics table where logs are sent. | | Option 15 | tenant_id [*required*] | string | Azure AD tenant ID. | | Option 15 | type [*required*] | enum | The destination type. The value should always be `microsoft_sentinel`. Allowed enum values: `microsoft_sentinel` | | destinations | Option 16 | object | The `new_relic` destination sends logs to the New Relic platform. | **Supported pipeline types:** logs | | Option 16 | account_id_key | string | Name of the environment variable or secret that holds the New Relic account ID. | | Option 16 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 16 | id [*required*] | string | The unique identifier for this component. | | Option 16 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 16 | license_key_key | string | Name of the environment variable or secret that holds the New Relic license key. | | Option 16 | region [*required*] | enum | The New Relic region. Allowed enum values: `us,eu` | | Option 16 | type [*required*] | enum | The destination type. The value should always be `new_relic`. Allowed enum values: `new_relic` | | destinations | Option 17 | object | The `opensearch` destination writes logs to an OpenSearch cluster. | **Supported pipeline types:** logs | | Option 17 | auth | object | Authentication settings for the Elasticsearch destination. When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets. | | auth | password_key | string | Name of the environment variable or secret that holds the Elasticsearch password (used when `strategy` is `basic`). | | auth | strategy [*required*] | enum | The authentication strategy to use. Allowed enum values: `basic,aws` | | auth | username_key | string | Name of the environment variable or secret that holds the Elasticsearch username (used when `strategy` is `basic`). | | Option 17 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 17 | bulk_index | string | The index to write logs to. | | Option 17 | data_stream | object | Configuration options for writing to OpenSearch Data Streams instead of a fixed index. | | data_stream | dataset | string | The data stream dataset for your logs. This groups logs by their source or application. | | data_stream | dtype | string | The data stream type for your logs. This determines how logs are categorized within the data stream. | | data_stream | namespace | string | The data stream namespace for your logs. This separates logs into different environments or domains. | | Option 17 | endpoint_url_key | string | Name of the environment variable or secret that holds the OpenSearch endpoint URL. | | Option 17 | id [*required*] | string | The unique identifier for this component. | | Option 17 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 17 | type [*required*] | enum | The destination type. The value should always be `opensearch`. Allowed enum values: `opensearch` | | destinations | Option 18 | object | The `rsyslog` destination forwards logs to an external `rsyslog` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 18 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 18 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog server endpoint URL. | | Option 18 | id [*required*] | string | The unique identifier for this component. | | Option 18 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 18 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 18 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 18 | type [*required*] | enum | The destination type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | destinations | Option 19 | object | The `sentinel_one` destination sends logs to SentinelOne. | **Supported pipeline types:** logs | | Option 19 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 19 | id [*required*] | string | The unique identifier for this component. | | Option 19 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 19 | region [*required*] | enum | The SentinelOne region to send logs to. Allowed enum values: `us,eu,ca,data_set_us` | | Option 19 | token_key | string | Name of the environment variable or secret that holds the SentinelOne API token. | | Option 19 | type [*required*] | enum | The destination type. The value should always be `sentinel_one`. Allowed enum values: `sentinel_one` | | destinations | Option 20 | object | The `socket` destination sends logs over TCP or UDP to a remote server. | **Supported pipeline types:** logs | | Option 20 | address_key | string | Name of the environment variable or secret that holds the socket address (host:port). | | Option 20 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 20 | encoding [*required*] | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 20 | framing [*required*] | | Framing method configuration. | | framing | Option 1 | object | Each log event is delimited by a newline character. | | Option 1 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod` object. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Event data is not delimited at all. | | Option 2 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod` object. Allowed enum values: `bytes` | | framing | Option 3 | object | Each log event is separated using the specified delimiter character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used as a delimiter. | | Option 3 | method [*required*] | enum | The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod` object. Allowed enum values: `character_delimited` | | Option 20 | id [*required*] | string | The unique identifier for this component. | | Option 20 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 20 | mode [*required*] | enum | Protocol used to send logs. Allowed enum values: `tcp,udp` | | Option 20 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 20 | type [*required*] | enum | The destination type. The value should always be `socket`. Allowed enum values: `socket` | | destinations | Option 21 | object | The `splunk_hec` destination forwards logs to Splunk using the HTTP Event Collector (HEC). | **Supported pipeline types:** logs | | Option 21 | auto_extract_timestamp | boolean | If `true`, Splunk tries to extract timestamps from incoming log events. If `false`, Splunk assigns the time the event was received. | | Option 21 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 21 | encoding | enum | Encoding format for log events. Allowed enum values: `json,raw_message` | | Option 21 | endpoint_url_key | string | Name of the environment variable or secret that holds the Splunk HEC endpoint URL. | | Option 21 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 21 | index | string | Optional name of the Splunk index where logs are written. | | Option 21 | indexed_fields | [string] | List of log field names to send as indexed fields to Splunk HEC. Available only when `encoding` is `json`. | | Option 21 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 21 | sourcetype | string | The Splunk sourcetype to assign to log events. | | Option 21 | token_key | string | Name of the environment variable or secret that holds the Splunk HEC token. | | Option 21 | type [*required*] | enum | The destination type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | destinations | Option 22 | object | The `sumo_logic` destination forwards logs to Sumo Logic. | **Supported pipeline types:** logs | | Option 22 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 22 | encoding | enum | The output encoding format. Allowed enum values: `json,raw_message,logfmt` | | Option 22 | endpoint_url_key | string | Name of the environment variable or secret that holds the Sumo Logic HTTP endpoint URL. | | Option 22 | header_custom_fields | [object] | A list of custom headers to include in the request to Sumo Logic. | | header_custom_fields | name [*required*] | string | The header field name. | | header_custom_fields | value [*required*] | string | The header field value. | | Option 22 | header_host_name | string | Optional override for the host name header. | | Option 22 | header_source_category | string | Optional override for the source category header. | | Option 22 | header_source_name | string | Optional override for the source name header. | | Option 22 | id [*required*] | string | The unique identifier for this component. | | Option 22 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 22 | type [*required*] | enum | The destination type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | destinations | Option 23 | object | The `syslog_ng` destination forwards logs to an external `syslog-ng` server over TCP or UDP using the syslog protocol. | **Supported pipeline types:** logs | | Option 23 | buffer | | Configuration for buffer settings on destination components. | | buffer | Option 1 | object | Options for configuring a disk buffer. | | Option 1 | max_size [*required*] | int64 | Maximum size of the disk buffer. | | Option 1 | type | enum | The type of the buffer that will be configured, a disk buffer. Allowed enum values: `disk` | | Option 1 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 2 | object | Options for configuring a memory buffer by byte size. | | Option 2 | max_size [*required*] | int64 | Maximum size of the memory buffer. | | Option 2 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 2 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | buffer | Option 3 | object | Options for configuring a memory buffer by queue length. | | Option 3 | max_events [*required*] | int64 | Maximum events for the memory buffer. | | Option 3 | type | enum | The type of the buffer that will be configured, a memory buffer. Allowed enum values: `memory` | | Option 3 | when_full | enum | Behavior when the buffer is full (block and stop accepting new events, or drop new events) Allowed enum values: `block,drop_newest` | | Option 23 | endpoint_url_key | string | Name of the environment variable or secret that holds the syslog-ng server endpoint URL. | | Option 23 | id [*required*] | string | The unique identifier for this component. | | Option 23 | inputs [*required*] | [string] | A list of component IDs whose output is used as the `input` for this component. | | Option 23 | keepalive | int64 | Optional socket keepalive duration in milliseconds. | | Option 23 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 23 | type [*required*] | enum | The destination type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | destinations | Option 24 | object | The `datadog_metrics` destination forwards metrics to Datadog. | **Supported pipeline types:** metrics | | Option 24 | id [*required*] | string | The unique identifier for this component. | | Option 24 | inputs [*required*] | [string] | A list of component IDs whose output is used as the input for this component. | | Option 24 | type [*required*] | enum | The destination type. The value should always be `datadog_metrics`. Allowed enum values: `datadog_metrics` | | config | pipeline_type | enum | The type of data being ingested. Defaults to `logs` if not specified. Allowed enum values: `logs,metrics` | | config | processor_groups | [object] | A list of processor groups that transform or enrich log data. | | processor_groups | display_name | string | The display name for a component. | | processor_groups | enabled [*required*] | boolean | Whether this processor group is enabled. | | processor_groups | id [*required*] | string | The unique identifier for the processor group. | | processor_groups | include [*required*] | string | Conditional expression for when this processor group should execute. | | processor_groups | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processor_groups | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | processors | [object] | **DEPRECATED**: A list of processor groups that transform or enrich log data. | **Deprecated:** This field is deprecated, you should now use the processor_groups field. | | processors | display_name | string | The display name for a component. | | processors | enabled [*required*] | boolean | Whether this processor group is enabled. | | processors | id [*required*] | string | The unique identifier for the processor group. | | processors | include [*required*] | string | Conditional expression for when this processor group should execute. | | processors | inputs [*required*] | [string] | A list of IDs for components whose output is used as the input for this processor group. | | processors | processors [*required*] | [ ] | Processors applied sequentially within this group. Events flow through each processor in order. | | processors | Option 1 | object | The `filter` processor allows conditional processing of logs/metrics based on a Datadog search query. Logs/metrics that match the `include` query are passed through; others are discarded. | **Supported pipeline types:** logs, metrics | | Option 1 | display_name | string | The display name for a component. | | Option 1 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | include [*required*] | string | A Datadog search query used to determine which logs/metrics should pass through the filter. Logs/metrics that match this query continue to downstream components; others are dropped. | | Option 1 | type [*required*] | enum | The processor type. The value should always be `filter`. Allowed enum values: `filter` | | processors | Option 2 | object | The `add_env_vars` processor adds environment variable values to log events. | **Supported pipeline types:** logs | | Option 2 | display_name | string | The display name for a component. | | Option 2 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used to reference this processor in the pipeline. | | Option 2 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 2 | type [*required*] | enum | The processor type. The value should always be `add_env_vars`. Allowed enum values: `add_env_vars` | | Option 2 | variables [*required*] | [object] | A list of environment variable mappings to apply to log fields. | | variables | field [*required*] | string | The target field in the log event. | | variables | name [*required*] | string | The name of the environment variable to read. | | processors | Option 3 | object | The `add_fields` processor adds static key-value fields to logs. | **Supported pipeline types:** logs | | Option 3 | display_name | string | The display name for a component. | | Option 3 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 3 | fields [*required*] | [object] | A list of static fields (key-value pairs) that is added to each log event processed by this component. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 3 | type [*required*] | enum | The processor type. The value should always be `add_fields`. Allowed enum values: `add_fields` | | processors | Option 4 | object | The `add_hostname` processor adds the hostname to log events. | **Supported pipeline types:** logs | | Option 4 | display_name | string | The display name for a component. | | Option 4 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 4 | type [*required*] | enum | The processor type. The value should always be `add_hostname`. Allowed enum values: `add_hostname` | | processors | Option 5 | object | The `custom_processor` processor transforms events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/) scripts with advanced filtering capabilities. | **Supported pipeline types:** logs | | Option 5 | display_name | string | The display name for a component. | | Option 5 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 5 | id [*required*] | string | The unique identifier for this processor. | | Option 5 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. This field should always be set to `*` for the custom_processor processor. | | Option 5 | remaps [*required*] | [object] | Array of VRL remap rules. | | remaps | drop_on_error [*required*] | boolean | Whether to drop events that caused errors during processing. | | remaps | enabled | boolean | Whether this remap rule is enabled. | | remaps | include [*required*] | string | A Datadog search query used to filter events for this specific remap rule. | | remaps | name [*required*] | string | A descriptive name for this remap rule. | | remaps | source [*required*] | string | The VRL script source code that defines the processing logic. | | Option 5 | type [*required*] | enum | The processor type. The value should always be `custom_processor`. Allowed enum values: `custom_processor` | | processors | Option 6 | object | The `datadog_tags` processor includes or excludes specific Datadog tags in your logs. | **Supported pipeline types:** logs | | Option 6 | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | Option 6 | display_name | string | The display name for a component. | | Option 6 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 6 | keys [*required*] | [string] | A list of tag keys. | | Option 6 | mode [*required*] | enum | The processing mode. Allowed enum values: `filter` | | Option 6 | type [*required*] | enum | The processor type. The value should always be `datadog_tags`. Allowed enum values: `datadog_tags` | | processors | Option 7 | object | The `dedupe` processor removes duplicate fields in log events. | **Supported pipeline types:** logs | | Option 7 | cache | object | Configuration for the cache used to detect duplicates. | | cache | num_events [*required*] | int64 | The number of events to cache for duplicate detection. | | Option 7 | display_name | string | The display name for a component. | | Option 7 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 7 | fields [*required*] | [string] | A list of log field paths to check for duplicates. | | Option 7 | id [*required*] | string | The unique identifier for this processor. | | Option 7 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 7 | mode [*required*] | enum | The deduplication mode to apply to the fields. Allowed enum values: `match,ignore` | | Option 7 | type [*required*] | enum | The processor type. The value should always be `dedupe`. Allowed enum values: `dedupe` | | processors | Option 8 | object | The `enrichment_table` processor enriches logs using a static CSV file, GeoIP database, or reference table. Exactly one of `file`, `geoip`, or `reference_table` must be configured. | **Supported pipeline types:** logs | | Option 8 | display_name | string | The display name for a component. | | Option 8 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 8 | file | object | Defines a static enrichment table loaded from a CSV file. | | file | encoding [*required*] | object | File encoding format. | | encoding | delimiter [*required*] | string | The `encoding` `delimiter`. | | encoding | includes_headers [*required*] | boolean | The `encoding` `includes_headers`. | | encoding | type [*required*] | enum | Specifies the encoding format (e.g., CSV) used for enrichment tables. Allowed enum values: `csv` | | file | key [*required*] | [object] | Key fields used to look up enrichment values. | | key | column [*required*] | string | The `items` `column`. | | key | comparison [*required*] | enum | Defines how to compare key fields for enrichment table lookups. Allowed enum values: `equals` | | key | field [*required*] | string | The `items` `field`. | | file | path [*required*] | string | Path to the CSV file. | | file | schema [*required*] | [object] | Schema defining column names and their types. | | schema | column [*required*] | string | The `items` `column`. | | schema | type [*required*] | enum | Declares allowed data types for enrichment table columns. Allowed enum values: `string,boolean,integer,float,date,timestamp` | | Option 8 | geoip | object | Uses a GeoIP database to enrich logs based on an IP field. | | geoip | key_field [*required*] | string | Path to the IP field in the log. | | geoip | locale [*required*] | string | Locale used to resolve geographical names. | | geoip | path [*required*] | string | Path to the GeoIP database file. | | Option 8 | id [*required*] | string | The unique identifier for this processor. | | Option 8 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 8 | reference_table | object | Uses a Datadog reference table to enrich logs. | | reference_table | app_key_key | string | Name of the environment variable or secret that holds the Datadog application key used to access the reference table. | | reference_table | columns | [string] | List of column names to include from the reference table. If not provided, all columns are included. | | reference_table | key_field [*required*] | string | Path to the field in the log event to match against the reference table. | | reference_table | table_id [*required*] | string | The unique identifier of the reference table. | | Option 8 | target [*required*] | string | Path where enrichment results should be stored in the log. | | Option 8 | type [*required*] | enum | The processor type. The value should always be `enrichment_table`. Allowed enum values: `enrichment_table` | | processors | Option 9 | object | The `generate_datadog_metrics` processor creates custom metrics from logs and sends them to Datadog. Metrics can be counters, gauges, or distributions and optionally grouped by log fields. | **Supported pipeline types:** logs | | Option 9 | display_name | string | The display name for a component. | | Option 9 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 9 | include | string | A Datadog search query used to determine which logs this processor targets. | | Option 9 | metrics | [object] | Configuration for generating individual metrics. | | metrics | group_by | [string] | Optional fields used to group the metric series. | | metrics | include [*required*] | string | Datadog filter query to match logs for metric generation. | | metrics | metric_type [*required*] | enum | Type of metric to create. Allowed enum values: `count,gauge,distribution` | | metrics | name [*required*] | string | Name of the custom metric to be created. | | metrics | value [*required*] | | Specifies how the value of the generated metric is computed. | | value | Option 1 | object | Strategy that increments a generated metric by one for each matching event. | | Option 1 | strategy [*required*] | enum | Increments the metric by 1 for each matching event. Allowed enum values: `increment_by_one` | | value | Option 2 | object | Strategy that increments a generated metric based on the value of a log field. | | Option 2 | field [*required*] | string | Name of the log field containing the numeric value to increment the metric by. | | Option 2 | strategy [*required*] | enum | Uses a numeric field in the log event as the metric increment. Allowed enum values: `increment_by_field` | | Option 9 | type [*required*] | enum | The processor type. Always `generate_datadog_metrics`. Allowed enum values: `generate_datadog_metrics` | | processors | Option 10 | object | The `ocsf_mapper` processor transforms logs into the OCSF schema using a predefined mapping configuration. | **Supported pipeline types:** logs | | Option 10 | display_name | string | The display name for a component. | | Option 10 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used to reference this component in other parts of the pipeline. | | Option 10 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 10 | keep_unmatched | boolean | Whether to keep an event that does not match any of the mapping filters. | | Option 10 | mappings [*required*] | [object] | A list of mapping rules to convert events to the OCSF format. | | mappings | include [*required*] | string | A Datadog search query used to select the logs that this mapping should apply to. | | mappings | mapping [*required*] | | Defines a single mapping rule for transforming logs into the OCSF schema. | | mapping | Option 1 | enum | Predefined library mappings for common log formats. Allowed enum values: `CloudTrail Account Change,GCP Cloud Audit CreateBucket,GCP Cloud Audit CreateSink,GCP Cloud Audit SetIamPolicy,GCP Cloud Audit UpdateSink,Github Audit Log API Activity,Google Workspace Admin Audit addPrivilege,Microsoft 365 Defender Incident,Microsoft 365 Defender UserLoggedIn,Okta System Log Authentication` | | mapping | Option 2 | object | Custom OCSF mapping configuration for transforming logs. | | Option 2 | mapping [*required*] | [object] | A list of field mapping rules for transforming log fields to OCSF schema fields. | | mapping | default | | The default value to use if the source field is missing or empty. | | mapping | dest [*required*] | string | The destination OCSF field path. | | mapping | lookup | object | Lookup table configuration for mapping source values to destination values. | | mapping | source | | The source field path from the log event. | | mapping | sources | | Multiple source field paths for combined mapping. | | mapping | value | | A static value to use for the destination field. | | Option 2 | metadata [*required*] | object | Metadata for the custom OCSF mapping. | | metadata | class [*required*] | string | The OCSF event class name. | | metadata | profiles | [string] | A list of OCSF profiles to apply. | | metadata | version [*required*] | string | The OCSF schema version. | | Option 2 | version [*required*] | int64 | The version of the custom mapping configuration. | | Option 10 | type [*required*] | enum | The processor type. The value should always be `ocsf_mapper`. Allowed enum values: `ocsf_mapper` | | processors | Option 11 | object | The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns. | **Supported pipeline types:** logs | | Option 11 | disable_library_rules | boolean | If set to `true`, disables the default Grok rules provided by Datadog. | | Option 11 | display_name | string | The display name for a component. | | Option 11 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 11 | id [*required*] | string | A unique identifier for this processor. | | Option 11 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 11 | rules [*required*] | [object] | The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied. | | rules | match_rules [*required*] | [object] | A list of Grok parsing rules that define how to extract fields from the source field. Each rule must contain a name and a valid Grok pattern. | | match_rules | name [*required*] | string | The name of the rule. | | match_rules | rule [*required*] | string | The definition of the Grok rule. | | rules | source [*required*] | string | The name of the field in the log event to apply the Grok rules to. | | rules | support_rules | [object] | A list of Grok helper rules that can be referenced by the parsing rules. | | support_rules | name [*required*] | string | The name of the Grok helper rule. | | support_rules | rule [*required*] | string | The definition of the Grok helper rule. | | Option 11 | type [*required*] | enum | The processor type. The value should always be `parse_grok`. Allowed enum values: `parse_grok` | | processors | Option 12 | object | The `parse_json` processor extracts JSON from a specified field and flattens it into the event. This is useful when logs contain embedded JSON as a string. | **Supported pipeline types:** logs | | Option 12 | display_name | string | The display name for a component. | | Option 12 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 12 | field [*required*] | string | The name of the log field that contains a JSON string. | | Option 12 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 12 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 12 | type [*required*] | enum | The processor type. The value should always be `parse_json`. Allowed enum values: `parse_json` | | processors | Option 13 | object | The `parse_xml` processor parses XML from a specified field and extracts it into the event. | **Supported pipeline types:** logs | | Option 13 | always_use_text_key | boolean | Whether to always use a text key for element content. | | Option 13 | attr_prefix | string | The prefix to use for XML attributes in the parsed output. | | Option 13 | display_name | string | The display name for a component. | | Option 13 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 13 | field [*required*] | string | The name of the log field that contains an XML string. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 13 | include_attr | boolean | Whether to include XML attributes in the parsed output. | | Option 13 | parse_bool | boolean | Whether to parse boolean values from strings. | | Option 13 | parse_null | boolean | Whether to parse null values. | | Option 13 | parse_number | boolean | Whether to parse numeric values from strings. | | Option 13 | text_key | string | The key name to use for text content within XML elements. Must be at least 1 character if specified. | | Option 13 | type [*required*] | enum | The processor type. The value should always be `parse_xml`. Allowed enum values: `parse_xml` | | processors | Option 14 | object | The `quota` processor measures logging traffic for logs that match a specified filter. When the configured daily quota is met, the processor can drop or alert. | **Supported pipeline types:** logs | | Option 14 | display_name | string | The display name for a component. | | Option 14 | drop_events | boolean | If set to `true`, logs that match the quota filter and are sent after the quota is exceeded are dropped. Logs that do not match the filter continue through the pipeline. **Note**: You can set either `drop_events` or `overflow_action`, but not both. | | Option 14 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | ignore_when_missing_partitions | boolean | If `true`, the processor skips quota checks when partition fields are missing from the logs. | | Option 14 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 14 | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | name [*required*] | string | Name of the quota. | | Option 14 | overflow_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | overrides | [object] | A list of alternate quota rules that apply to specific sets of events, identified by matching field values. Each override can define a custom limit. | | overrides | fields [*required*] | [object] | A list of field matchers used to apply a specific override. If an event matches all listed key-value pairs, the corresponding override limit is enforced. | | fields | name [*required*] | string | The field name. | | fields | value [*required*] | string | The field value. | | overrides | limit [*required*] | object | The maximum amount of data or number of events allowed before the quota is enforced. Can be specified in bytes or events. | | limit | enforce [*required*] | enum | Unit for quota enforcement in bytes for data size or events for count. Allowed enum values: `bytes,events` | | limit | limit [*required*] | int64 | The limit for quota enforcement. | | Option 14 | partition_fields | [string] | A list of fields used to segment log traffic for quota enforcement. Quotas are tracked independently by unique combinations of these field values. | | Option 14 | too_many_buckets_action | enum | The action to take when the quota or bucket limit is exceeded. Options: | | Option 14 | type [*required*] | enum | The processor type. The value should always be `quota`. Allowed enum values: `quota` | | processors | Option 15 | object | The `reduce` processor aggregates and merges logs based on matching keys and merge strategies. | **Supported pipeline types:** logs | | Option 15 | display_name | string | The display name for a component. | | Option 15 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 15 | group_by [*required*] | [string] | A list of fields used to group log events for merging. | | Option 15 | id [*required*] | string | The unique identifier for this processor. | | Option 15 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 15 | merge_strategies [*required*] | [object] | List of merge strategies defining how values from grouped events should be combined. | | merge_strategies | path [*required*] | string | The field path in the log event. | | merge_strategies | strategy [*required*] | enum | The merge strategy to apply. Allowed enum values: `discard,retain,sum,max,min,array,concat,concat_newline,concat_raw,shortest_array` | | Option 15 | type [*required*] | enum | The processor type. The value should always be `reduce`. Allowed enum values: `reduce` | | processors | Option 16 | object | The `remove_fields` processor deletes specified fields from logs. | **Supported pipeline types:** logs | | Option 16 | display_name | string | The display name for a component. | | Option 16 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 16 | fields [*required*] | [string] | A list of field names to be removed from each log event. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 16 | type [*required*] | enum | The processor type. The value should always be `remove_fields`. Allowed enum values: `remove_fields` | | processors | Option 17 | object | The `rename_fields` processor changes field names. | **Supported pipeline types:** logs | | Option 17 | display_name | string | The display name for a component. | | Option 17 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 17 | fields [*required*] | [object] | A list of rename rules specifying which fields to rename in the event, what to rename them to, and whether to preserve the original fields. | | fields | destination [*required*] | string | The field name to assign the renamed value to. | | fields | preserve_source [*required*] | boolean | Indicates whether the original field, that is received from the source, should be kept (`true`) or removed (`false`) after renaming. | | fields | source [*required*] | string | The original field name in the log event that should be renamed. | | Option 17 | id [*required*] | string | A unique identifier for this component. Used to reference this component in other parts of the pipeline (e.g., as input to downstream components). | | Option 17 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 17 | type [*required*] | enum | The processor type. The value should always be `rename_fields`. Allowed enum values: `rename_fields` | | processors | Option 18 | object | The `sample` processor allows probabilistic sampling of logs at a fixed rate. | **Supported pipeline types:** logs | | Option 18 | display_name | string | The display name for a component. | | Option 18 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 18 | group_by | [string] | Optional list of fields to group events by. Each group is sampled independently. | | Option 18 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 18 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 18 | percentage [*required*] | double | The percentage of logs to sample. | | Option 18 | type [*required*] | enum | The processor type. The value should always be `sample`. Allowed enum values: `sample` | | processors | Option 19 | object | The `sensitive_data_scanner` processor detects and optionally redacts sensitive data in log events. | **Supported pipeline types:** logs | | Option 19 | display_name | string | The display name for a component. | | Option 19 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 19 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 19 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 19 | rules [*required*] | [object] | A list of rules for identifying and acting on sensitive data patterns. | | rules | keyword_options | object | Configuration for keywords used to reinforce sensitive data pattern detection. | | keyword_options | keywords [*required*] | [string] | A list of keywords to match near the sensitive pattern. | | keyword_options | proximity [*required*] | int64 | Maximum number of tokens between a keyword and a sensitive value match. | | rules | name [*required*] | string | A name identifying the rule. | | rules | on_match [*required*] | | Defines what action to take when sensitive data is matched. | | on_match | Option 1 | object | Configuration for completely redacting matched sensitive data. | | Option 1 | action [*required*] | enum | Action type that completely replaces the matched sensitive data with a fixed replacement string to remove all visibility. Allowed enum values: `redact` | | Option 1 | options [*required*] | object | Configuration for fully redacting sensitive data. | | options | replace [*required*] | string | The string used to replace matched sensitive data (for example, "***" or "[REDACTED]"). | | on_match | Option 2 | object | Configuration for hashing matched sensitive values. | | Option 2 | action [*required*] | enum | Action type that replaces the matched sensitive data with a hashed representation, preserving structure while securing content. Allowed enum values: `hash` | | Option 2 | options | object | Optional settings for the hash action. When omitted or empty, matched sensitive data is replaced with a deterministic hashed value that preserves structure for analytics while protecting the original content. Reserved for future hash configuration (for example, algorithm or salt). | | on_match | Option 3 | object | Configuration for partially redacting matched sensitive data. | | Option 3 | action [*required*] | enum | Action type that redacts part of the sensitive data while preserving a configurable number of characters, typically used for masking purposes (e.g., show last 4 digits of a credit card). Allowed enum values: `partial_redact` | | Option 3 | options [*required*] | object | Controls how partial redaction is applied, including character count and direction. | | options | characters [*required*] | int64 | Number of characters to leave visible from the start or end of the matched value; the rest are redacted. | | options | direction [*required*] | enum | Indicates whether to redact characters from the first or last part of the matched value. Allowed enum values: `first,last` | | rules | pattern [*required*] | | Pattern detection configuration for identifying sensitive data using either a custom regex or a library reference. | | pattern | Option 1 | object | Defines a custom regex-based pattern for identifying sensitive data in logs. | | Option 1 | options [*required*] | object | Options for defining a custom regex pattern. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | rule [*required*] | string | A regular expression used to detect sensitive values. Must be a valid regex. | | Option 1 | type [*required*] | enum | Indicates a custom regular expression is used for matching. Allowed enum values: `custom` | | pattern | Option 2 | object | Specifies a pattern from Datadog's sensitive data detection library to match known sensitive data types. | | Option 2 | options [*required*] | object | Options for selecting a predefined library pattern and enabling keyword support. | | options | description | string | Human-readable description providing context about a sensitive data scanner rule | | options | id [*required*] | string | Identifier for a predefined pattern from the sensitive data scanner pattern library. | | options | use_recommended_keywords | boolean | Whether to augment the pattern with recommended keywords (optional). | | Option 2 | type [*required*] | enum | Indicates that a predefined library pattern is used. Allowed enum values: `library` | | rules | scope [*required*] | | Determines which parts of the log the pattern-matching rule should be applied to. | | scope | Option 1 | object | Includes only specific fields for sensitive data scanning. | | Option 1 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 1 | target [*required*] | enum | Applies the rule only to included fields. Allowed enum values: `include` | | scope | Option 2 | object | Excludes specific fields from sensitive data scanning. | | Option 2 | options [*required*] | object | Fields to which the scope rule applies. | | options | fields [*required*] | [string] | List of log attribute names (field paths) to which the scope applies. Only these fields are included in or excluded from pattern matching. | | Option 2 | target [*required*] | enum | Excludes specific fields from processing. Allowed enum values: `exclude` | | scope | Option 3 | object | Applies scanning across all available fields. | | Option 3 | target [*required*] | enum | Applies the rule to all fields. Allowed enum values: `all` | | rules | tags | [string] | Tags assigned to this rule for filtering and classification. | | Option 19 | type [*required*] | enum | The processor type. The value should always be `sensitive_data_scanner`. Allowed enum values: `sensitive_data_scanner` | | processors | Option 20 | object | The `split_array` processor splits array fields into separate events based on configured rules. | **Supported pipeline types:** logs | | Option 20 | arrays [*required*] | [object] | A list of array split configurations. | | arrays | field [*required*] | string | The path to the array field to split. | | arrays | include [*required*] | string | A Datadog search query used to determine which logs this array split operation targets. | | Option 20 | display_name | string | The display name for a component. | | Option 20 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 20 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 20 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. For split_array, this should typically be `*`. | | Option 20 | type [*required*] | enum | The processor type. The value should always be `split_array`. Allowed enum values: `split_array` | | processors | Option 21 | object | The `throttle` processor limits the number of events that pass through over a given time window. | **Supported pipeline types:** logs | | Option 21 | display_name | string | The display name for a component. | | Option 21 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 21 | group_by | [string] | Optional list of fields used to group events before the threshold has been reached. | | Option 21 | id [*required*] | string | The unique identifier for this processor. | | Option 21 | include [*required*] | string | A Datadog search query used to determine which logs this processor targets. | | Option 21 | threshold [*required*] | int64 | the number of events allowed in a given time window. Events sent after the threshold has been reached, are dropped. | | Option 21 | type [*required*] | enum | The processor type. The value should always be `throttle`. Allowed enum values: `throttle` | | Option 21 | window [*required*] | double | The time window in seconds over which the threshold applies. | | processors | Option 22 | object | The `metric_tags` processor filters metrics based on their tags using Datadog tag key patterns. | **Supported pipeline types:** metrics | | Option 22 | display_name | string | The display name for a component. | | Option 22 | enabled [*required*] | boolean | Indicates whether the processor is enabled. | | Option 22 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 22 | include [*required*] | string | A Datadog search query that determines which metrics the processor targets. | | Option 22 | rules [*required*] | [object] | A list of rules for filtering metric tags. | | rules | action [*required*] | enum | The action to take on tags with matching keys. Allowed enum values: `include,exclude` | | rules | include [*required*] | string | A Datadog search query used to determine which metrics this rule targets. | | rules | keys [*required*] | [string] | A list of tag keys to include or exclude. | | rules | mode [*required*] | enum | The processing mode for tag filtering. Allowed enum values: `filter` | | Option 22 | type [*required*] | enum | The processor type. The value should always be `metric_tags`. Allowed enum values: `metric_tags` | | config | sources [*required*] | [ ] | A list of configured data sources for the pipeline. | | sources | Option 1 | object | The `datadog_agent` source collects logs/metrics from the Datadog Agent. | **Supported pipeline types:** logs, metrics | | Option 1 | address_key | string | Name of the environment variable or secret that holds the listen address for the Datadog Agent source. | | Option 1 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 1 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 1 | type [*required*] | enum | The source type. The value should always be `datadog_agent`. Allowed enum values: `datadog_agent` | | sources | Option 2 | object | The `amazon_data_firehose` source ingests logs from AWS Data Firehose. | **Supported pipeline types:** logs | | Option 2 | address_key | string | Name of the environment variable or secret that holds the Firehose delivery stream address. | | Option 2 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 2 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 2 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 2 | type [*required*] | enum | The source type. The value should always be `amazon_data_firehose`. Allowed enum values: `amazon_data_firehose` | | sources | Option 3 | object | The `amazon_s3` source ingests logs from an Amazon S3 bucket. It supports AWS authentication and TLS encryption. | **Supported pipeline types:** logs | | Option 3 | auth | object | AWS authentication credentials used for accessing AWS services such as S3. If omitted, the system's default credentials are used (for example, the IAM role and environment variables). | | auth | assume_role | string | The Amazon Resource Name (ARN) of the role to assume. | | auth | external_id | string | A unique identifier for cross-account role assumption. | | auth | session_name | string | A session identifier used for logging and tracing the assumed role session. | | Option 3 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 3 | region [*required*] | string | AWS region where the S3 bucket resides. | | Option 3 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 3 | type [*required*] | enum | The source type. Always `amazon_s3`. Allowed enum values: `amazon_s3` | | Option 3 | url_key | string | Name of the environment variable or secret that holds the S3 bucket URL. | | sources | Option 4 | object | The `fluent_bit` source ingests logs from Fluent Bit. | **Supported pipeline types:** logs | | Option 4 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent Bit receiver. | | Option 4 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 4 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 4 | type [*required*] | enum | The source type. The value should always be `fluent_bit`. Allowed enum values: `fluent_bit` | | sources | Option 5 | object | The `fluentd` source ingests logs from a Fluentd-compatible service. | **Supported pipeline types:** logs | | Option 5 | address_key | string | Name of the environment variable or secret that holds the listen address for the Fluent receiver. | | Option 5 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 5 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 5 | type [*required*] | enum | The source type. The value should always be `fluentd. Allowed enum values: `fluentd` | | sources | Option 6 | object | The `google_pubsub` source ingests logs from a Google Cloud Pub/Sub subscription. | **Supported pipeline types:** logs | | Option 6 | auth | object | Google Cloud credentials used to authenticate with Google Cloud Storage. | | auth | credentials_file [*required*] | string | Path to the Google Cloud service account key file. | | Option 6 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 6 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 6 | project [*required*] | string | The Google Cloud project ID that owns the Pub/Sub subscription. | | Option 6 | subscription [*required*] | string | The Pub/Sub subscription name from which messages are consumed. | | Option 6 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 6 | type [*required*] | enum | The source type. The value should always be `google_pubsub`. Allowed enum values: `google_pubsub` | | sources | Option 7 | object | The `http_client` source scrapes logs from HTTP endpoints at regular intervals. | **Supported pipeline types:** logs | | Option 7 | auth_strategy | enum | Optional authentication strategy for HTTP requests. Allowed enum values: `none,basic,bearer,custom` | | Option 7 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 7 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 7 | endpoint_url_key | string | Name of the environment variable or secret that holds the HTTP endpoint URL to scrape. | | Option 7 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 7 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`). | | Option 7 | scrape_interval_secs | int64 | The interval (in seconds) between HTTP scrape requests. | | Option 7 | scrape_timeout_secs | int64 | The timeout (in seconds) for each scrape request. | | Option 7 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 7 | token_key | string | Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`). | | Option 7 | type [*required*] | enum | The source type. The value should always be `http_client`. Allowed enum values: `http_client` | | Option 7 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`). | | sources | Option 8 | object | The `http_server` source collects logs over HTTP POST from external services. | **Supported pipeline types:** logs | | Option 8 | address_key | string | Name of the environment variable or secret that holds the listen address for the HTTP server. | | Option 8 | auth_strategy [*required*] | enum | HTTP authentication method. Allowed enum values: `none,plain` | | Option 8 | custom_key | string | Name of the environment variable or secret that holds a custom header value (used with custom auth strategies). | | Option 8 | decoding [*required*] | enum | The decoding format used to interpret incoming logs. Allowed enum values: `bytes,gelf,json,syslog` | | Option 8 | id [*required*] | string | Unique ID for the HTTP server source. | | Option 8 | password_key | string | Name of the environment variable or secret that holds the password (used when `auth_strategy` is `plain`). | | Option 8 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 8 | type [*required*] | enum | The source type. The value should always be `http_server`. Allowed enum values: `http_server` | | Option 8 | username_key | string | Name of the environment variable or secret that holds the username (used when `auth_strategy` is `plain`). | | sources | Option 9 | object | The `kafka` source ingests data from Apache Kafka topics. | **Supported pipeline types:** logs | | Option 9 | bootstrap_servers_key | string | Name of the environment variable or secret that holds the Kafka bootstrap servers list. | | Option 9 | group_id [*required*] | string | Consumer group ID used by the Kafka client. | | Option 9 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 9 | librdkafka_options | [object] | Optional list of advanced Kafka client configuration options, defined as key-value pairs. | | librdkafka_options | name [*required*] | string | The name of the `librdkafka` configuration option to set. | | librdkafka_options | value [*required*] | string | The value assigned to the specified `librdkafka` configuration option. | | Option 9 | sasl | object | Specifies the SASL mechanism for authenticating with a Kafka cluster. | | sasl | mechanism | enum | SASL mechanism used for Kafka authentication. Allowed enum values: `PLAIN,SCRAM-SHA-256,SCRAM-SHA-512` | | sasl | password_key | string | Name of the environment variable or secret that holds the SASL password. | | sasl | username_key | string | Name of the environment variable or secret that holds the SASL username. | | Option 9 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 9 | topics [*required*] | [string] | A list of Kafka topic names to subscribe to. The source ingests messages from each topic specified. | | Option 9 | type [*required*] | enum | The source type. The value should always be `kafka`. Allowed enum values: `kafka` | | sources | Option 10 | object | The `logstash` source ingests logs from a Logstash forwarder. | **Supported pipeline types:** logs | | Option 10 | address_key | string | Name of the environment variable or secret that holds the listen address for the Logstash receiver. | | Option 10 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 10 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 10 | type [*required*] | enum | The source type. The value should always be `logstash`. Allowed enum values: `logstash` | | sources | Option 11 | object | The `rsyslog` source listens for logs over TCP or UDP from an `rsyslog` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 11 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog receiver. | | Option 11 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 11 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 11 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 11 | type [*required*] | enum | The source type. The value should always be `rsyslog`. Allowed enum values: `rsyslog` | | sources | Option 12 | object | The `socket` source ingests logs over TCP or UDP. | **Supported pipeline types:** logs | | Option 12 | address_key | string | Name of the environment variable or secret that holds the listen address for the socket. | | Option 12 | framing [*required*] | | Framing method configuration for the socket source. | | framing | Option 1 | object | Byte frames which are delimited by a newline character. | | Option 1 | method [*required*] | enum | Byte frames which are delimited by a newline character. Allowed enum values: `newline_delimited` | | framing | Option 2 | object | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). | | Option 2 | method [*required*] | enum | Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments). Allowed enum values: `bytes` | | framing | Option 3 | object | Byte frames which are delimited by a chosen character. | | Option 3 | delimiter [*required*] | string | A single ASCII character used to delimit events. | | Option 3 | method [*required*] | enum | Byte frames which are delimited by a chosen character. Allowed enum values: `character_delimited` | | framing | Option 4 | object | Byte frames according to the octet counting format as per RFC6587. | | Option 4 | method [*required*] | enum | Byte frames according to the octet counting format as per RFC6587. Allowed enum values: `octet_counting` | | framing | Option 5 | object | Byte frames which are chunked GELF messages. | | Option 5 | method [*required*] | enum | Byte frames which are chunked GELF messages. Allowed enum values: `chunked_gelf` | | Option 12 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 12 | mode [*required*] | enum | Protocol used to receive logs. Allowed enum values: `tcp,udp` | | Option 12 | tls | object | TLS configuration. Relevant only when `mode` is `tcp`. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 12 | type [*required*] | enum | The source type. The value should always be `socket`. Allowed enum values: `socket` | | sources | Option 13 | object | The `splunk_hec` source implements the Splunk HTTP Event Collector (HEC) API. | **Supported pipeline types:** logs | | Option 13 | address_key | string | Name of the environment variable or secret that holds the listen address for the HEC API. | | Option 13 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 13 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 13 | type [*required*] | enum | The source type. Always `splunk_hec`. Allowed enum values: `splunk_hec` | | sources | Option 14 | object | The `splunk_tcp` source receives logs from a Splunk Universal Forwarder over TCP. TLS is supported for secure transmission. | **Supported pipeline types:** logs | | Option 14 | address_key | string | Name of the environment variable or secret that holds the listen address for the Splunk TCP receiver. | | Option 14 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 14 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 14 | type [*required*] | enum | The source type. Always `splunk_tcp`. Allowed enum values: `splunk_tcp` | | sources | Option 15 | object | The `sumo_logic` source receives logs from Sumo Logic collectors. | **Supported pipeline types:** logs | | Option 15 | address_key | string | Name of the environment variable or secret that holds the listen address for the Sumo Logic receiver. | | Option 15 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 15 | type [*required*] | enum | The source type. The value should always be `sumo_logic`. Allowed enum values: `sumo_logic` | | sources | Option 16 | object | The `syslog_ng` source listens for logs over TCP or UDP from a `syslog-ng` server using the syslog protocol. | **Supported pipeline types:** logs | | Option 16 | address_key | string | Name of the environment variable or secret that holds the listen address for the syslog-ng receiver. | | Option 16 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 16 | mode [*required*] | enum | Protocol used by the syslog source to receive messages. Allowed enum values: `tcp,udp` | | Option 16 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 16 | type [*required*] | enum | The source type. The value should always be `syslog_ng`. Allowed enum values: `syslog_ng` | | sources | Option 17 | object | The `opentelemetry` source receives telemetry data using the OpenTelemetry Protocol (OTLP) over gRPC and HTTP. | **Supported pipeline types:** logs, metrics | | Option 17 | grpc_address_key | string | Environment variable name containing the gRPC server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | http_address_key | string | Environment variable name containing the HTTP server address for receiving OTLP data. Must be a valid environment variable name (alphanumeric characters and underscores only). | | Option 17 | id [*required*] | string | The unique identifier for this component. Used in other parts of the pipeline to reference this component (for example, as the `input` to downstream components). | | Option 17 | tls | object | Configuration for enabling TLS encryption between the pipeline component and external services. | | tls | ca_file | string | Path to the Certificate Authority (CA) file used to validate the server's TLS certificate. | | tls | crt_file [*required*] | string | Path to the TLS client certificate file used to authenticate the pipeline component with upstream or downstream services. | | tls | key_file | string | Path to the private key file associated with the TLS client certificate. Used for mutual TLS authentication. | | tls | key_pass_key | string | Name of the environment variable or secret that holds the passphrase for the private key file. | | Option 17 | type [*required*] | enum | The source type. The value should always be `opentelemetry`. Allowed enum values: `opentelemetry` | | config | use_legacy_search_syntax | boolean | Set to `true` to continue using the legacy search syntax while migrating filter queries. After migrating all queries to the new syntax, set to `false`. The legacy syntax is deprecated and will eventually be removed. Requires Observability Pipelines Worker 2.11 or later. See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax/) for more information. | | attributes | name [*required*] | string | Name of the pipeline. | | data | type [*required*] | string | The resource type identifier. For pipeline resources, this should always be set to `pipelines`. | {% /tab %} {% tab title="Example" %} ##### ```json { "data": { "attributes": { "config": { "pipeline_type": "metrics", "destinations": [ { "id": "datadog-metrics-destination", "inputs": [ "my-processor-group" ], "type": "datadog_metrics" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "*", "inputs": [ "opentelemetry-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "env:production", "type": "filter" } ] } ], "sources": [ { "id": "opentelemetry-source", "type": "opentelemetry" } ] }, "name": "Metrics OTel Pipeline" }, "type": "pipelines" } } ``` ##### ```json { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Main Observability Pipeline" }, "type": "pipelines" } } ``` ##### ```json { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "ocsf-mapper-processor", "include": "service:my-service", "mappings": [ { "include": "source:custom", "mapping": { "mapping": [ { "default": "", "dest": "time", "source": "timestamp" }, { "default": "", "dest": "severity", "source": "level" }, { "default": "", "dest": "device.type", "lookup": { "table": [ { "contains": "Desktop", "value": "desktop" } ] }, "source": "host.type" } ], "metadata": { "class": "Device Inventory Info", "profiles": [ "container" ], "version": "1.3.0" }, "version": 1 } } ], "type": "ocsf_mapper" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "OCSF Custom Mapper Pipeline" }, "type": "pipelines" } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} Response containing validation errors. | Parent field | Field | Type | Description | | ------------ | ------------------------- | -------- | ---------------------------------------------------------------------------------------------- | | | errors | [object] | The `ValidationResponse` `errors`. | | errors | meta [*required*] | object | Describes additional metadata for validation errors, including field names and error messages. | | meta | field | string | The field name that caused the error. | | meta | id | string | The ID of the component in which the error occurred. | | meta | message [*required*] | string | The detailed error message. | | errors | title [*required*] | string | A short, human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "meta": { "field": "region", "id": "datadog-agent-source", "message": "Field 'region' is required" }, "title": "Field 'region' is required" } ] } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Not Authorized {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/validate" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "config": { "pipeline_type": "metrics", "destinations": [ { "id": "datadog-metrics-destination", "inputs": [ "my-processor-group" ], "type": "datadog_metrics" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "*", "inputs": [ "opentelemetry-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "env:production", "type": "filter" } ] } ], "sources": [ { "id": "opentelemetry-source", "type": "opentelemetry" } ] }, "name": "Metrics OTel Pipeline" }, "type": "pipelines" } } EOF ##### \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/validate" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "filter-processor", "include": "status:error", "type": "filter" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "Main Observability Pipeline" }, "type": "pipelines" } } EOF ##### \# Curl commandcurl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/obs-pipelines/pipelines/validate" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "data": { "attributes": { "config": { "destinations": [ { "id": "datadog-logs-destination", "inputs": [ "my-processor-group" ], "type": "datadog_logs" } ], "processor_groups": [ { "enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": [ "datadog-agent-source" ], "processors": [ { "enabled": true, "id": "ocsf-mapper-processor", "include": "service:my-service", "mappings": [ { "include": "source:custom", "mapping": { "mapping": [ { "default": "", "dest": "time", "source": "timestamp" }, { "default": "", "dest": "severity", "source": "level" }, { "default": "", "dest": "device.type", "lookup": { "table": [ { "contains": "Desktop", "value": "desktop" } ] }, "source": "host.type" } ], "metadata": { "class": "Device Inventory Info", "profiles": [ "container" ], "version": "1.3.0" }, "version": 1 } } ], "type": "ocsf_mapper" } ] } ], "sources": [ { "id": "datadog-agent-source", "type": "datadog_agent" } ] }, "name": "OCSF Custom Mapper Pipeline" }, "type": "pipelines" } } EOF ##### ```go // Validate an observability pipeline returns "OK" response package main import ( "context" "encoding/json" "fmt" "os" "github.com/DataDog/datadog-api-client-go/v2/api/datadog" "github.com/DataDog/datadog-api-client-go/v2/api/datadogV2" ) func main() { body := datadogV2.ObservabilityPipelineSpec{ Data: datadogV2.ObservabilityPipelineSpecData{ Attributes: datadogV2.ObservabilityPipelineDataAttributes{ Config: datadogV2.ObservabilityPipelineConfig{ Destinations: []datadogV2.ObservabilityPipelineConfigDestinationItem{ datadogV2.ObservabilityPipelineConfigDestinationItem{ ObservabilityPipelineDatadogLogsDestination: &datadogV2.ObservabilityPipelineDatadogLogsDestination{ Id: "datadog-logs-destination", Inputs: []string{ "my-processor-group", }, Type: datadogV2.OBSERVABILITYPIPELINEDATADOGLOGSDESTINATIONTYPE_DATADOG_LOGS, }}, }, ProcessorGroups: []datadogV2.ObservabilityPipelineConfigProcessorGroup{ { Enabled: true, Id: "my-processor-group", Include: "service:my-service", Inputs: []string{ "datadog-agent-source", }, Processors: []datadogV2.ObservabilityPipelineConfigProcessorItem{ datadogV2.ObservabilityPipelineConfigProcessorItem{ ObservabilityPipelineFilterProcessor: &datadogV2.ObservabilityPipelineFilterProcessor{ Enabled: true, Id: "filter-processor", Include: "status:error", Type: datadogV2.OBSERVABILITYPIPELINEFILTERPROCESSORTYPE_FILTER, }}, }, }, }, Sources: []datadogV2.ObservabilityPipelineConfigSourceItem{ datadogV2.ObservabilityPipelineConfigSourceItem{ ObservabilityPipelineDatadogAgentSource: &datadogV2.ObservabilityPipelineDatadogAgentSource{ Id: "datadog-agent-source", Type: datadogV2.OBSERVABILITYPIPELINEDATADOGAGENTSOURCETYPE_DATADOG_AGENT, }}, }, }, Name: "Main Observability Pipeline", }, Type: "pipelines", }, } ctx := datadog.NewDefaultContext(context.Background()) configuration := datadog.NewConfiguration() apiClient := datadog.NewAPIClient(configuration) api := datadogV2.NewObservabilityPipelinesApi(apiClient) resp, r, err := api.ValidatePipeline(ctx, body) if err != nil { fmt.Fprintf(os.Stderr, "Error when calling `ObservabilityPipelinesApi.ValidatePipeline`: %v\n", err) fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) } responseContent, _ := json.MarshalIndent(resp, "", " ") fmt.Fprintf(os.Stdout, "Response from `ObservabilityPipelinesApi.ValidatePipeline`:\n%s\n", responseContent) } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=go) and then save the example to `main.go` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" go run "main.go" ##### ```java // Validate an observability pipeline returns "OK" response import com.datadog.api.client.ApiClient; import com.datadog.api.client.ApiException; import com.datadog.api.client.v2.api.ObservabilityPipelinesApi; import com.datadog.api.client.v2.model.ObservabilityPipelineConfig; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigDestinationItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorGroup; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigProcessorItem; import com.datadog.api.client.v2.model.ObservabilityPipelineConfigSourceItem; import com.datadog.api.client.v2.model.ObservabilityPipelineDataAttributes; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSource; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogAgentSourceType; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestination; import com.datadog.api.client.v2.model.ObservabilityPipelineDatadogLogsDestinationType; import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessor; import com.datadog.api.client.v2.model.ObservabilityPipelineFilterProcessorType; import com.datadog.api.client.v2.model.ObservabilityPipelineSpec; import com.datadog.api.client.v2.model.ObservabilityPipelineSpecData; import com.datadog.api.client.v2.model.ValidationResponse; import java.util.Collections; public class Example { public static void main(String[] args) { ApiClient defaultClient = ApiClient.getDefaultApiClient(); ObservabilityPipelinesApi apiInstance = new ObservabilityPipelinesApi(defaultClient); ObservabilityPipelineSpec body = new ObservabilityPipelineSpec() .data( new ObservabilityPipelineSpecData() .attributes( new ObservabilityPipelineDataAttributes() .config( new ObservabilityPipelineConfig() .destinations( Collections.singletonList( new ObservabilityPipelineConfigDestinationItem( new ObservabilityPipelineDatadogLogsDestination() .id("datadog-logs-destination") .inputs( Collections.singletonList( "my-processor-group")) .type( ObservabilityPipelineDatadogLogsDestinationType .DATADOG_LOGS)))) .processorGroups( Collections.singletonList( new ObservabilityPipelineConfigProcessorGroup() .enabled(true) .id("my-processor-group") .include("service:my-service") .inputs( Collections.singletonList( "datadog-agent-source")) .processors( Collections.singletonList( new ObservabilityPipelineConfigProcessorItem( new ObservabilityPipelineFilterProcessor() .enabled(true) .id("filter-processor") .include("status:error") .type( ObservabilityPipelineFilterProcessorType .FILTER)))))) .sources( Collections.singletonList( new ObservabilityPipelineConfigSourceItem( new ObservabilityPipelineDatadogAgentSource() .id("datadog-agent-source") .type( ObservabilityPipelineDatadogAgentSourceType .DATADOG_AGENT))))) .name("Main Observability Pipeline")) .type("pipelines")); try { ValidationResponse result = apiInstance.validatePipeline(body); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ObservabilityPipelinesApi#validatePipeline"); System.err.println("Status code: " + e.getCode()); System.err.println("Reason: " + e.getResponseBody()); System.err.println("Response headers: " + e.getResponseHeaders()); e.printStackTrace(); } } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=java) and then save the example to `Example.java` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" java "Example.java" ##### ```python """ Validate an observability pipeline returns "OK" response """ from datadog_api_client import ApiClient, Configuration from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig from datadog_api_client.v2.model.observability_pipeline_config_processor_group import ( ObservabilityPipelineConfigProcessorGroup, ) from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import ( ObservabilityPipelineDatadogAgentSource, ) from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import ( ObservabilityPipelineDatadogAgentSourceType, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import ( ObservabilityPipelineDatadogLogsDestination, ) from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import ( ObservabilityPipelineDatadogLogsDestinationType, ) from datadog_api_client.v2.model.observability_pipeline_filter_processor import ObservabilityPipelineFilterProcessor from datadog_api_client.v2.model.observability_pipeline_filter_processor_type import ( ObservabilityPipelineFilterProcessorType, ) from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData body = ObservabilityPipelineSpec( data=ObservabilityPipelineSpecData( attributes=ObservabilityPipelineDataAttributes( config=ObservabilityPipelineConfig( destinations=[ ObservabilityPipelineDatadogLogsDestination( id="datadog-logs-destination", inputs=[ "my-processor-group", ], type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS, ), ], processor_groups=[ ObservabilityPipelineConfigProcessorGroup( enabled=True, id="my-processor-group", include="service:my-service", inputs=[ "datadog-agent-source", ], processors=[ ObservabilityPipelineFilterProcessor( enabled=True, id="filter-processor", include="status:error", type=ObservabilityPipelineFilterProcessorType.FILTER, ), ], ), ], sources=[ ObservabilityPipelineDatadogAgentSource( id="datadog-agent-source", type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT, ), ], ), name="Main Observability Pipeline", ), type="pipelines", ), ) configuration = Configuration() with ApiClient(configuration) as api_client: api_instance = ObservabilityPipelinesApi(api_client) response = api_instance.validate_pipeline(body=body) print(response) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=python) and then save the example to `example.py` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" python3 "example.py" ##### ```ruby # Validate an observability pipeline returns "OK" response require "datadog_api_client" api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({ data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({ attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({ config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({ destinations: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({ id: "datadog-logs-destination", inputs: [ "my-processor-group", ], type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, }), ], processor_groups: [ DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({ enabled: true, id: "my-processor-group", include: "service:my-service", inputs: [ "datadog-agent-source", ], processors: [ DatadogAPIClient::V2::ObservabilityPipelineFilterProcessor.new({ enabled: true, id: "filter-processor", include: "status:error", type: DatadogAPIClient::V2::ObservabilityPipelineFilterProcessorType::FILTER, }), ], }), ], sources: [ DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({ id: "datadog-agent-source", type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, }), ], }), name: "Main Observability Pipeline", }), type: "pipelines", }), }) p api_instance.validate_pipeline(body) ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=ruby) and then save the example to `example.rb` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" rb "example.rb" ##### ```rust // Validate an observability pipeline returns "OK" response use datadog_api_client::datadog; use datadog_api_client::datadogV2::api_observability_pipelines::ObservabilityPipelinesAPI; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfig; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigDestinationItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorGroup; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigProcessorItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineConfigSourceItem; use datadog_api_client::datadogV2::model::ObservabilityPipelineDataAttributes; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSource; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogAgentSourceType; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestination; use datadog_api_client::datadogV2::model::ObservabilityPipelineDatadogLogsDestinationType; use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessor; use datadog_api_client::datadogV2::model::ObservabilityPipelineFilterProcessorType; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpec; use datadog_api_client::datadogV2::model::ObservabilityPipelineSpecData; #[tokio::main] async fn main() { let body = ObservabilityPipelineSpec::new( ObservabilityPipelineSpecData::new( ObservabilityPipelineDataAttributes::new( ObservabilityPipelineConfig::new( vec![ ObservabilityPipelineConfigDestinationItem::ObservabilityPipelineDatadogLogsDestination( Box::new( ObservabilityPipelineDatadogLogsDestination::new( "datadog-logs-destination".to_string(), vec!["my-processor-group".to_string()], ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS, ), ), ) ], vec![ ObservabilityPipelineConfigSourceItem::ObservabilityPipelineDatadogAgentSource( Box::new( ObservabilityPipelineDatadogAgentSource::new( "datadog-agent-source".to_string(), ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT, ), ), ) ], ).processor_groups( vec![ ObservabilityPipelineConfigProcessorGroup::new( true, "my-processor-group".to_string(), "service:my-service".to_string(), vec!["datadog-agent-source".to_string()], vec![ ObservabilityPipelineConfigProcessorItem::ObservabilityPipelineFilterProcessor( Box::new( ObservabilityPipelineFilterProcessor::new( true, "filter-processor".to_string(), "status:error".to_string(), ObservabilityPipelineFilterProcessorType::FILTER, ), ), ) ], ) ], ), "Main Observability Pipeline".to_string(), ), "pipelines".to_string(), ), ); let configuration = datadog::Configuration::new(); let api = ObservabilityPipelinesAPI::with_config(configuration); let resp = api.validate_pipeline(body).await; if let Ok(value) = resp { println!("{:#?}", value); } else { println!("{:#?}", resp.unwrap_err()); } } ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=rust) and then save the example to `src/main.rs` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" cargo run ##### ```typescript /** * Validate an observability pipeline returns "OK" response */ import { client, v2 } from "@datadog/datadog-api-client"; const configuration = client.createConfiguration(); const apiInstance = new v2.ObservabilityPipelinesApi(configuration); const params: v2.ObservabilityPipelinesApiValidatePipelineRequest = { body: { data: { attributes: { config: { destinations: [ { id: "datadog-logs-destination", inputs: ["my-processor-group"], type: "datadog_logs", }, ], processorGroups: [ { enabled: true, id: "my-processor-group", include: "service:my-service", inputs: ["datadog-agent-source"], processors: [ { enabled: true, id: "filter-processor", include: "status:error", type: "filter", }, ], }, ], sources: [ { id: "datadog-agent-source", type: "datadog_agent", }, ], }, name: "Main Observability Pipeline", }, type: "pipelines", }, }, }; apiInstance .validatePipeline(params) .then((data: v2.ValidationResponse) => { console.log( "API called successfully. Returned data: " + JSON.stringify(data) ); }) .catch((error: any) => console.error(error)); ``` #### Instructions First [install the library and its dependencies](https://docs.datadoghq.com/api/latest/?code-lang=typescript) and then save the example to `example.ts` and run following commands: DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comddog-gov.com" DD_API_KEY="" DD_APP_KEY="" tsc "example.ts" {% /tab %}