Environment Variables

This product is not supported for your selected Datadog site. ().

Overview

Some Observability Pipelines components require setting up environment variables. This document lists the environments variables for the different sources, processors, and destinations.

Component environment variables

    Amazon Data Firehose

    • Amazon Data Firehose address:
      • The Observability Pipelines Worker listens to this socket address to receive logs from Amazon Data Firehose.
      • The address is stored in the environment variable DD_OP_SOURCE_AWS_DATA_FIREHOSE_ADDRESS.
    • Amazon Data Firehose TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_AWS_DATA_FIREHOSE_KEY_PASS.

    Amazon S3

    • Amazon S3 SQS URL:
      • The URL of the SQS queue to which the S3 bucket sends the notification events.
      • Stored in the environment variable: DD_OP_SOURCE_AWS_S3_SQS_URL
    • AWS_CONFIG_FILE path:
      • The path to the AWS configuration file local to this node.
      • Stored in the environment variable AWS_CONFIG_FILE.
    • AWS_PROFILE name:
      • The name of the profile to use within these files.
      • Stored in the environment variable AWS_PROFILE.
    • AWS S3 TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_AWS_S3_KEY_PASS.

    Datadog Agent

    • Datadog Agent address:
      • The Observability Pipelines Worker listens to this socket address to receive logs from the Datadog Agent.
      • Stored in the environment variable DD_OP_SOURCE_DATADOG_AGENT_ADDRESS.
    • Datadog Agent TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_DATADOG_AGENT_KEY_PASS.

    Fluent

    • Fluent socket address and port:
      • The Observability Pipelines Worker listens on this address for incoming log messages.
      • Stored in the environment variable DD_OP_SOURCE_FLUENT_ADDRESS.
    • Fluent Bit TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_FLUENT_KEY_PASS.

    Google Pub/Sub

    There are no environment variables for the Google Pub/Sub source.

    HTTP Client

    • HTTP/s endpoint URL:
      • The Observability Pipelines Worker collects log events from this endpoint. For example, https://127.0.0.8/logs.
      • Stored in the environment variable DD_OP_SOURCE_HTTP_CLIENT_ENDPOINT_URL.
    • HTTP/S Client TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_HTTP_CLIENT_KEY_PASS.
    • If you are using basic authentication:
      • HTTP/S endpoint authentication username and password.
      • Stored in the environment variables DD_OP_SOURCE_HTTP_CLIENT_USERNAME and DD_OP_SOURCE_HTTP_CLIENT_PASSWORD.
    • If you are using bearer authentication:
      • HTTP/S endpoint bearer token.
      • Stored in the environment variable DD_OP_SOURCE_HTTP_CLIENT_BEARER_TOKEN.

    HTTP Server

    • HTTP/S server address:
      • The Observability Pipelines Worker listens to this socket address, such as 0.0.0.0:9997, for your HTTP client logs.
      • Stored in the environment variable DD_OP_SOURCE_HTTP_SERVER_ADDRESS.

    Kafka

    • The host and port of the Kafka bootstrap servers.
      • The bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. The host and port must be entered in the format of host:port, such as 10.14.22.123:9092. If there is more than one server, use commas to separate them.
      • Stored in the environment variable DD_OP_SOURCE_KAFKA_BOOTSTRAP_SERVERS.
    • SASL (when enabled):
      • Kafka SASL username
        • Stored in the environment variable DD_OP_SOURCE_KAFKA_SASL_USERNAME.
      • Kafka SASL password
        • Stored in the environment variable DD_OP_SOURCE_KAFKA_SASL_PASSWORD.
    • Kafka TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_KAFKA_KEY_PASS.

    Logstash

    • Logstash address and port:
      • The Observability Pipelines Worker listens on this address, such as 0.0.0.0:9997, for incoming log messages.
      • Stored in the environment variable as DD_OP_SOURCE_LOGSTASH_ADDRESS
    • Logstash TLS passphrase:
      • Stored in the environment variable DD_OP_SOURCE_LOGSTASH_KEY_PASS.

    OpenTelemetry

    You must provide both HTTP and gRPC endpoints. Configure your OTLP exporters to point to one of these endpoints. See Send logs to the Observability Pipelines Worker for more information.

    • HTTP listener address

      • The Observability Pipelines Worker listens to this socket address to receive logs from the OTel collector.
      • Stored in the environment variable DD_OP_SOURCE_OTEL_HTTP_ADDRESS.

    • gRPC listener address

      • The Observability Pipelines Worker listens to this socket address to receive logs from the OTel collector.
      • Stored in the environment variable DD_OP_SOURCE_OTEL_GRPC_ADDRESS.

    If TLS is enabled:

    • OpenTelemetry TLS passphrase
      • Stored in the environment variable DD_OP_SOURCE_OTEL_KEY_PASS.

    Socket

    • Socket address:

      • The address and port where the Observability Pipelines Worker listens for incoming logs.
      • Stored in the environment variable DD_OP_SOURCE_SOCKET_ADDRESS.

    • TLS passphrase (when enabled):

      • Stored in the environment variable DD_OP_SOURCE_SOCKET_KEY_PASS.

    Splunk HEC

    • Splunk HEC address:
      • The bind address that your Observability Pipelines Worker listens on to receive logs originally intended for the Splunk indexer. For example, 0.0.0.0:8088
        Note: /services/collector/event is automatically appended to the endpoint.
      • Stored in the environment variable DD_OP_SOURCE_SPLUNK_HEC_ADDRESS.
    • Splunk HEC TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_SPLUNK_HEC_KEY_PASS.

    Splunk TCP

    • Splunk TCP address:
      • The Observability Pipelines Worker listens to this socket address to receive logs from the Splunk Forwarder. For example, 0.0.0.0:9997.
      • Stored in the environment variable DD_OP_SOURCE_SPLUNK_TCP_ADDRESS.
    • Splunk TCP TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_SPLUNK_TCP_KEY_PASS.

    Sumo Logic

    • Sumo Logic address:
      • The bind address that your Observability Pipelines Worker listens on to receive logs originally intended for the Sumo Logic HTTP Source. For example, 0.0.0.0:80.
        Note: /receiver/v1/http/ path is automatically appended to the endpoint.
      • Stored in the environment variable DD_OP_SOURCE_SUMO_LOGIC_ADDRESS.

    Syslog

    • rsyslog or syslog-ng address:
      • The Observability Pipelines Worker listens on this bind address to receive logs from the Syslog forwarder. For example, 0.0.0.0:9997.
      • Stored in the environment variable DD_OP_SOURCE_SYSLOG_ADDRESS.
    • rsyslog or syslog-ng TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_SOURCE_SYSLOG_KEY_PASS.

    Add environment variables

    • Allowlist
      • The allowlist is a comma-separated list of environment variables you want to pull values from and use with this processor.
      • Stored in the environment variable DD_OP_PROCESSOR_ADD_ENV_VARS_ALLOWLIST.

    Amazon OpenSearch

    • Amazon OpenSearch authentication username:
      • Stored in the environment variable DD_OP_DESTINATION_AMAZON_OPENSEARCH_USERNAME.
    • Amazon OpenSearch authentication password:
      • Stored in the environment variable DD_OP_DESTINATION_AMAZON_OPENSEARCH_PASSWORD.
    • Amazon OpenSearch endpoint URL:
      • Stored in the environment variable DD_OP_DESTINATION_AMAZON_OPENSEARCH_ENDPOINT_URL.

    Amazon Security Lake

    • Amazon Security Lake TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_DESTINATION_AMAZON_SECURITY_LAKE_KEY_PASS.

    Chronicle

    • Google Chronicle endpoint URL:
      • Stored in the environment variable DD_OP_DESTINATION_GOOGLE_CHRONICLE_UNSTRUCTURED_ENDPOINT_URL.

    CrowdStrike NG-SIEM

    • CrowdStrike HEC ingestion URL:
      • Note: Do not include the suffix /services/collector in the URL. The URL must follow this format: https://<your_instance_id>.ingest.us-1.crowdstrike.com.
      • Stored in the environment variable DD_OP_DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_ENDPOINT_URL.
    • CrowdStrike HEC API token:
      • Stored in the environment variable DD_OP_DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_TOKEN.
    • CrowdStrike Next-Gen SIEM HEC TLS passphrase:
      • Stored in the environment variable DD_OP_DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_KEY_PASS.

    Datadog

    No environment variables required.

    Datadog Archives

    Amazon S3

    There are no environment variables to configure.

    Google Cloud Storage

    There are no environment variables to configure.

    Azure Storage

    • Azure connections string to give the Worker access to your Azure Storage bucket.
      • Stored in the environment variable DD_OP_DESTINATION_DATADOG_ARCHIVES_AZURE_BLOB_CONNECTION_STRING.

    Elasticsearch

    • Elasticsearch authentication username:
      • Stored in the environment variable DD_OP_DESTINATION_ELASTICSEARCH_USERNAME.
    • Elasticsearch authentication password:
      • Stored in the environment variable DD_OP_DESTINATION_ELASTICSEARCH_PASSWORD.
    • Elasticsearch endpoint URL:
      • Stored in the environment variable DD_OP_DESTINATION_ELASTICSEARCH_ENDPOINT_URL.

    Kafka

    Kafka bootstrap servers

    • The host and port of the Kafka bootstrap servers.
    • This is the bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. The host and port must be entered in the format of host:port, such as 10.14.22.123:9092. If there is more than one server, use commas to separate them.
    • Stored in the environment: DD_OP_DESTINATION_KAFKA_BOOTSTRAP_SERVERS.

    TLS (when enabled)

    • If TLS is enabled, the Kafka TLS passphrase is needed.
    • Stored in the environment: DD_OP_DESTINATION_KAFKA_KEY_PASS.

    SASL (when enabled)

    • Kafka SASL username
      • Stored in the environment: DD_OP_DESTINATION_KAFKA_SASL_USERNAME.
    • Kafka SASL password
      • Stored in the environment: DD_OP_DESTINATION_KAFKA_SASL_PASSWORD.

    Google Pub/Sub

    By default the Worker sends data to the global endpoint: https://pubsub.googleapis.com.

    If your Pub/Sub topic is region-specific, configure the Google Pub/Sub alternative endpoint URL with the regional endpoint. See About Pub/Sub endpoints for more information.

    Stored in the environment variable DD_OP_DESTINATION_GCP_PUBSUB_ENDPOINT_URL.

    TLS (when enabled)

    • Google Pub/Sub TLS passphrase:
      • Stored in the environment variable DD_OP_DESTINATION_GCP_PUBSUB_KEY_PASS.

    HTTP Client

    • HTTP/S client URI endpoint:
      • Stored in the environment variable DD_OP_DESTINATION_HTTP_CLIENT_URI.
    • HTTP/S Client TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_DESTINATION_HTTP_CLIENT_KEY_PASS.
    • If you are using basic authentication:
      • HTTP/S endpoint authentication username and password.
      • Stored in the environment variable DD_OP_DESTINATION_HTTP_CLIENT_USERNAME and DD_OP_DESTINATION_HTTP_CLIENT_PASSWORD.
    • If you are using bearer authentication:
      • HTTP/S endpoint bearer token.
      • Stored in the environment variable DD_OP_DESTINATION_HTTP_CLIENT_BEARER_TOKEN.

    Microsoft Sentinel

    • Data collection endpoint (DCE)
      • The DCE endpoint URL is shown as the Logs Ingestion Endpoint or Data Collection Endpoint on the DCR Overview page. An example URL: https://<DCE-ID>.ingest.monitor.azure.com.
      • Stored in the environment variable DD_OP_DESTINATION_MICROSOFT_SENTINEL_DCE_URI
    • Client secret
      • This is the Azure AD application’s client secret, such as 550e8400-e29b-41d4-a716-446655440000.
      • Stored in the environment variable DD_OP_DESTINATION_MICROSOFT_SENTINEL_CLIENT_SECRET

    New Relic

    • New Relic account ID:
      • Stored in the environment variable DD_OP_DESTINATION_NEW_RELIC_ACCOUNT_ID.
    • New Relic license:
      • Stored in the environment variable DD_OP_DESTINATION_NEW_RELIC_LICENSE_KEY.

    OpenSearch

    • OpenSearch authentication username:
      • Stored in the environment variable DD_OP_DESTINATION_OPENSEARCH_USERNAME.
    • OpenSearch authentication password:
      • Stored in the environment variable DD_OP_DESTINATION_OPENSEARCH_PASSWORD.
    • OpenSearch endpoint URL:
      • Stored in the environment variable DD_OP_DESTINATION_OPENSEARCH_ENDPOINT_URL.

    SentinelOne

    • SentinelOne write access token:
      • Stored in the environment variable DD_OP_DESTINATION_SENTINEL_ONE_TOKEN.

    Socket

    • Socket address:
      • The address to which the Observability Pipelines Worker sends processed logs.
      • Stored in the environment variable DD_OP_DESTINATION_SOCKET_ADDRESS.
    • TLS passphrase:
      • Stored in the environment variable DD_OP_DESTINATION_SOCKET_KEY_PASS.

    Splunk HEC

    • Splunk HEC token:
      • The Splunk HEC token for the Splunk indexer. Note: Depending on your shell and environment, you may not want to wrap your environment variable in quotes.
      • Stored in the environment variable DD_OP_DESTINATION_SPLUNK_HEC_TOKEN.
    • Base URL of the Splunk instance:
      • The Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed logs to. For example, https://hec.splunkcloud.com:8088.
        Note: /services/collector/event path is automatically appended to the endpoint.
      • Stored in the environment variable DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL.

    Sumo Logic

    • Unique URL generated for the HTTP Logs and Metrics Source to receive log data.
      • The Sumo Logic HTTP Source endpoint. The Observability Pipelines Worker sends processed logs to this endpoint. For example, https://<ENDPOINT>.collection.sumologic.com/receiver/v1/http/<UNIQUE_HTTP_COLLECTOR_CODE>, where:
        • <ENDPOINT> is your Sumo collection endpoint.
        • <UNIQUE_HTTP_COLLECTOR_CODE> is the string that follows the last forward slash (/) in the upload URL for the HTTP source.
      • Stored in the environment variable DD_OP_DESTINATION_SUMO_LOGIC_HTTP_COLLECTOR_URL.

    Syslog

    • The rsyslog or syslog-ng endpoint URL. For example, 127.0.0.1:9997.
      • The Observability Pipelines Worker sends logs to this address and port.
      • Stored in the environment variable DD_OP_DESTINATION_SYSLOG_ENDPOINT_URL.
    • The ryslog or syslog-ng TLS passphrase (when enabled):
      • Stored in the environment variable DD_OP_DESTINATION_SYSLOG_KEY_PASS.