Environment Variables

This page is not yet available in Spanish. We are working on its translation.
If you have any questions or feedback about our current translation project, feel free to reach out to us!

Overview

Some Observability Pipelines components require setting up environment variables. This document lists the environments variables for the different sources, processors, and destinations.

Component environment variables

    Amazon Data Firehose

    • Amazon Data Firehose address
      • The Observability Pipelines Worker listens to this socket address to receive logs from Amazon Data Firehose.
      • The address is stored in the environment variable AWS_DATA_FIREHOSE_ADDRESS.

    Amazon S3

    • Amazon S3 SQS URL
      • The URL of the SQS queue to which the S3 bucket sends the notification events.
      • Stored as the environment variable: DD_OP_SOURCE_AWS_S3_SQS_URL
    • AWS_CONFIG_FILE path
      • The path to the AWS configuration file local to this node.
      • Stored as the environment variable: AWS_CONFIG_FILE.
    • AWS_PROFILE name
      • The name of the profile to use within these files.
      • Stored as the environment variable: AWS_PROFILE.

    Datadog Agent

    • Datadog Agent address:
      • The Observability Pipelines Worker listens to this socket address to receive logs from the Datadog Agent.
      • Stored in the environment variableDD_OP_SOURCE_DATADOG_AGENT_ADDRESS.

    Fluent

    • Fluent socket address and port:
      • The Observability Pipelines Worker listens on this address for incoming log messages.
      • Stored in the environment variable DD_OP_SOURCE_FLUENT_ADDRESS.

    Google Pub/Sub

    There are no environment variables for the Google Pub/Sub source.

    HTTP Client

    • HTTP/s endpoint URL:
      • The Observability Pipelines Worker collects log events from this endpoint. For example, https://127.0.0.8/logs.
      • Stored as the environment variable: DD_OP_SOURCE_HTTP_CLIENT_ENDPOINT_URL.
    • If you are using basic authentication:
      • HTTP/S endpoint authentication username and password.
      • Stored as the environment variables: DD_OP_SOURCE_HTTP_CLIENT_USERNAME and DD_OP_SOURCE_HTTP_CLIENT_PASSWORD.
    • If you are using bearer authentication:
      • HTTP/S endpoint bearer token.
      • Stored as the environment variable: DD_OP_SOURCE_HTTP_CLIENT_BEARER_TOKEN.

    HTTP Server

    • HTTP/S server address:
      • The Observability Pipelines Worker listens to this socket address, such as 0.0.0.0:9997, for your HTTP client logs.
      • Stored in the environment variable: DD_OP_SOURCE_HTTP_SERVER_ADDRESS.

    Kafka

    • The host and port of the Kafka bootstrap servers.
      • The bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. The host and port must be entered in the format of host:port, such as 10.14.22.123:9092. If there is more than one server, use commas to separate them.
      • Stored as the environment variable: DD_OP_SOURCE_KAFKA_BOOTSTRAP_SERVERS.
    • If you enabled SASL:
      • Kafka SASL username
        • Stored as the environment variable: DD_OP_SOURCE_KAFKA_SASL_USERNAME.
      • Kafka SASL password
        • Stored as the environment variable: DD_OP_SOURCE_KAFKA_SASL_PASSWORD.

    Logstash

    • Logstash address and port:
      • The Observability Pipelines Worker listens on this address, such as 0.0.0.0:9997, for incoming log messages.
      • Stored in the environment variable as: DD_OP_SOURCE_LOGSTASH_ADDRESS

    Splunk HEC

    • Splunk HEC address:
      • The bind address that your Observability Pipelines Worker listens on to receive logs originally intended for the Splunk indexer. For example, 0.0.0.0:8088
        Note: /services/collector/event is automatically appended to the endpoint.
      • Stored in the environment variable DD_OP_SOURCE_SPLUNK_HEC_ADDRESS.

    Splunk TCP

    • Splunk TCP address:
      • The Observability Pipelines Worker listens to this socket address to receive logs from the Splunk Forwarder. For example, 0.0.0.0:9997.
      • Stored in the environment variable DD_OP_SOURCE_SPLUNK_TCP_ADDRESS.

    Sumo Logic

    • Sumo Logic address:
      • The bind address that your Observability Pipelines Worker listens on to receive logs originally intended for the Sumo Logic HTTP Source. For example, 0.0.0.0:80.
        Note: /receiver/v1/http/ path is automatically appended to the endpoint.
      • Stored in the environment variable DD_OP_SOURCE_SUMO_LOGIC_ADDRESS.

    Syslog

    • rsyslog or syslog-ng address:
      • The Observability Pipelines Worker listens on this bind address to receive logs from the Syslog forwarder. For example, 0.0.0.0:9997.
      • Stored in the environment variable DD_OP_SOURCE_SYSLOG_ADDRESS.

    Add environment variables

    • Allowlist
      • The allowlist is a comma-separated list of environment variables you want to pull values from and use with this processor.
      • Stored in the environment variable DD_OP_PROCESSOR_ADD_ENV_VARS_ALLOWLIST.

    Amazon OpenSearch

    • Amazon OpenSearch authentication username:
      • Stored in the environment variable: DD_OP_DESTINATION_AMAZON_OPENSEARCH_USERNAME.
    • Amazon OpenSearch authentication password:
      • Stored in the environment variable: DD_OP_DESTINATION_AMAZON_OPENSEARCH_PASSWORD.
    • Amazon OpenSearch endpoint URL:
      • Stored in the environment variable: DD_OP_DESTINATION_AMAZON_OPENSEARCH_ENDPOINT_URL.

    Chronicle

    • Google Chronicle endpoint URL:
      • Stored in the environment variable: DD_OP_DESTINATION_GOOGLE_CHRONICLE_UNSTRUCTURED_ENDPOINT_URL.

    CrowdStrike NG-SIEM

    • CrowdStrike HEC ingestion URL:

      • Stored in the environment variable DD_OP_DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_ENDPOINT_URL.

    • CrowdStrike HEC API token:

      • Stored in the environment variable DD_OP_DESTINATION_CROWDSTRIKE_NEXT_GEN_SIEM_TOKEN.

    Datadog

    No environment variables required.

    Datadog Archives

    Amazon S3

    • AWS access key ID of your S3 archive:

      • Stored in the environment variable: DD_OP_DESTINATION_DATADOG_ARCHIVES_AWS_ACCESS_KEY_ID

    • AWS secret access key ID of your S3 archive:

      • The AWS secret access key ID for the S3 archive bucket.
      • Stored in the environment variable DD_OP_DESTINATION_DATADOG_ARCHIVES_AWS_SECRET_KEY.

    Google Cloud Storage

    There are no environment variables to configure.

    Azure Storage

    • Azure connections string to give the Worker access to your Azure Storage bucket.
      • Stored in the environment variable: DD_OP_DESTINATION_DATADOG_ARCHIVES_AZURE_BLOB_CONNECTION_STRING.

    Elasticsearch

    • Elasticsearch authentication username:
      • Stored in the environment variable: DD_OP_DESTINATION_ELASTICSEARCH_USERNAME.
    • Elasticsearch authentication password:
      • Stored in the environment variable: DD_OP_DESTINATION_ELASTICSEARCH_PASSWORD.
    • Elasticsearch endpoint URL:
      • Stored in the environment variable: DD_OP_DESTINATION_ELASTICSEARCH_ENDPOINT_URL.

    Microsoft Sentinel

    • Data collection endpoint (DCE)
      • The DCE endpoint URL is shown as the Logs Ingestion Endpoint or Data Collection Endpoint on the DCR Overview page. An example URL: https://<DCE-ID>.ingest.monitor.azure.com/dataCollectionRules/<DCR-Immutable-ID>/streams/<Stream-Name>?api-version=2023-01-01.
      • Stored as the environment variable DD_OP_DESTINATION_MICROSOFT_SENTINEL_DCE_URI
    • Client secret
      • This is the Azure AD application’s client secret, such as 550e8400-e29b-41d4-a716-446655440000.
      • Stored as the environment variable DD_OP_DESTINATION_MICROSOFT_SENTINEL_CLIENT_SECRET

    New Relic

    • New Relic account ID:
      • Stored in the environment variable: DD_OP_DESTINATION_NEW_RELIC_ACCOUNT_ID.
    • New Relic license:
      • Stored in the environment variable: DD_OP_DESTINATION_NEW_RELIC_LICENSE_KEY.

    OpenSearch

    • OpenSearch authentication username:
      • Stored in the environment variable: DD_OP_DESTINATION_OPENSEARCH_USERNAME.
    • OpenSearch authentication password:
      • Stored in the environment variable: DD_OP_DESTINATION_OPENSEARCH_PASSWORD.
    • OpenSearch endpoint URL:
      • Stored in the environment variable: DD_OP_DESTINATION_OPENSEARCH_ENDPOINT_URL.

    SentinelOne

    • SentinelOne write access token:
      • Stored as the environment variable: DD_OP_DESTINATION_SENTINEL_ONE_TOKEN

    Splunk HEC

    • Splunk HEC token:
      • The Splunk HEC token for the Splunk indexer.
      • Stored in the environment variable DD_OP_DESTINATION_SPLUNK_HEC_TOKEN.
    • Base URL of the Splunk instance:
      • The Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed logs to. For example, https://hec.splunkcloud.com:8088.
        Note: /services/collector/event path is automatically appended to the endpoint.
      • Stored in the environment variable DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL.

    Sumo Logic

    • Unique URL generated for the HTTP Logs and Metrics Source to receive log data.
      • The Sumo Logic HTTP Source endpoint. The Observability Pipelines Worker sends processed logs to this endpoint. For example, https://<ENDPOINT>.collection.sumologic.com/receiver/v1/http/<UNIQUE_HTTP_COLLECTOR_CODE>, where:
        • <ENDPOINT> is your Sumo collection endpoint.
        • <UNIQUE_HTTP_COLLECTOR_CODE> is the string that follows the last forward slash (/) in the upload URL for the HTTP source.
      • Stored in the environment variable DD_OP_DESTINATION_SUMO_LOGIC_HTTP_COLLECTOR_URL.

    Syslog

    • The rsyslog or syslog-ng endpoint URL. For example, 127.0.0.1:9997.
      • The Observability Pipelines Worker sends logs to this address and port.
      • Stored as the environment variable: DD_OP_DESTINATION_SYSLOG_ENDPOINT_URL.