Splunk HTTP Event Collector (HEC) Destination

This product is not supported for your selected Datadog site. ().
Available for:

Logs

Use Observability Pipelines’ Splunk HTTP Event Collector (HEC) destination to send logs to Splunk HEC.

Setup

Set up the Splunk HEC destination and its environment variables when you set up a pipeline. The information below is configured in the pipelines UI.

Set up the destination

Observability Pipelines compresses logs with the gzip (level 6) algorithm.
Only enter the identifiers for the Splunk HEC token and endpoint. Do not enter the actual values.
  1. Enter the identifier for your token. If you leave it blank, the default is used.
  2. Enter the identifier for your endpoint URL. If you leave it blank, the default is used.

Optional settings

  1. Enter the name of the Splunk index you want your data in. This has to be an allowed index for your HEC. See template syntax if you want to route logs to different indexes based on specific fields in your logs.
  2. Select whether the timestamp should be auto-extracted. If set to true, Splunk extracts the timestamp from the message with the expected format of yyyy-mm-dd hh:mm:ss.
  3. Optionally, set the sourcetype to override Splunk’s default value, which is httpevent for HEC data. See template syntax if you want to route logs to different source types based on specific fields in your logs.
  4. Optionally, toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing logs to disk, ensuring buffered logs persist through a Worker restart. See Configurable buffers for destinations for more information.
    • If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
    • To configure a buffer on your destination:
      1. Select the buffer type you want to set (Memory or Disk).
      2. Enter the buffer size and select the unit.
        • Maximum memory buffer size is 128 GB.
        • Maximum disk buffer size is 500 GB.

Set secrets

These are the defaults used for secret identifiers and environment variables.

Note: If you enter identifiers for your secrets and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

    • Splunk HEC token identifier:
      • References the Splunk HEC token for the Splunk indexer.
      • The default identifier is DESTINATION_SPLUNK_HEC_TOKEN.
    • Splunk HEC endpoint URL identifier:
      • References the Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed logs to. For example, https://hec.splunkcloud.com:8088.
      • Note: /services/collector/event path is automatically appended to the endpoint.
      • The default identifier is DESTINATION_SPLUNK_HEC_ENDPOINT_URL.
    • Splunk HEC token:
      • The Splunk HEC token for the Splunk indexer. Note: Depending on your shell and environment, you may not want to wrap your environment variable in quotes.
      • The default environment variable is DD_OP_DESTINATION_SPLUNK_HEC_TOKEN.
    • Base URL of the Splunk instance:
      • The Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed logs to. For example, https://hec.splunkcloud.com:8088. Note: /services/collector/event path is automatically appended to the endpoint.
      • The default environment variable is DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL.

    How the destination works

    Event batching

    A batch of events is flushed when one of these parameters is met. See event batching for more information.

    Max EventsMax BytesTimeout (seconds)
    None1,000,0001