Splunk HTTP Event Collector (HEC) Destination

This product is not supported for your selected Datadog site. ().

Overview

Use Observability Pipelines’ Splunk HTTP Event Collector (HEC) destination to send metrics to Splunk HEC.

Setup

Configure the Splunk HEC destination when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The steps in this section are configured in the UI.

Notes:

  • The Splunk index you send your metrics to must be a metrics index. If you send your metrics to an events index, you can’t view them in Splunk using any metrics type queries, such as mcatalog and mstats.
  • If you don’t attach your index to the Splunk authentication token you are using for Observability Pipelines, you must enter the name of the index when you set up the destination.
For Secrets Management: Only enter the identifiers for the Splunk HEC token, endpoint, and if applicable, the TLS key pass. Do not enter the actual values.
If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP_. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

After you select the Splunk HEC destination in the pipeline UI:

  1. Enter the identifier for your token. If you leave it blank, the default is used.
  2. Enter the identifier for your endpoint URL. If you leave it blank, the default is used.

Optional settings

Default namespace

Enter a default namespace to use as a prefix for metrics that don’t already have a namespace. The namespace is prepended to the metric name with a period (.). For example, in service.cpu.usage, service is the namespace.

Compression

To compress your metrics with gzip, select gzip in the dropdown menu. The default compression is None.

Splunk index

Enter the name of the Splunk metrics index to which you are sending metrics. See template syntax to route metrics to different indexes based on specific fields.

Note: The Index field is optional only if you have an index attached to the Splunk authentication token you are using for Observability Pipelines. Otherwise, you must enter the name of the Splunk metrics index.

Source

Enter a value to add the source field to your metrics.

Source type override

Set the sourcetype to override Splunk’s default value, which is httpevent for HEC data. See template syntax to route metrics to different source types based on specific fields.

Buffering

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

  • If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
  • To configure a buffer on your destination:
    1. Select the buffer type you want to set (Memory or Disk).
    2. Enter the buffer size and select the unit.
      1. Maximum memory buffer size is 128 GB.
      2. Maximum disk buffer size is 500 GB.
    3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Enable TLS

Toggle the switch to Enable TLS.

  • If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
  • The following certificate and key files are required:
    • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
    • Notes:
      • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
      • The file must be readable by the observability-pipelines-worker group and user.

Secret defaults

These are the defaults used for secret identifiers and environment variables.

    • Splunk HEC token identifier:
      • References the Splunk HEC token for the Splunk indexer.
      • The default identifier is DESTINATION_SPLUNK_HEC_TOKEN.
    • Splunk HEC endpoint URL identifier:
      • References the Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed data to. For example, https://hec.splunkcloud.com:8088.
      • Note: /services/collector/event path is automatically appended to the endpoint.
      • The default identifier is DESTINATION_SPLUNK_HEC_ENDPOINT_URL.
    • Splunk HEC token:
      • The Splunk HEC token for the Splunk indexer. Note: Depending on your shell and environment, you may not want to wrap your environment variable in quotes.
      • The default environment variable is DD_OP_DESTINATION_SPLUNK_HEC_TOKEN.
    • Base URL of the Splunk instance:
      • The Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed data to. For example, https://hec.splunkcloud.com:8088. Note: /services/collector/event path is automatically appended to the endpoint.
      • The default environment variable is DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL.

    How the destination works

    Event batching

    A batch of events is flushed when one of the following parameters is met. See event batching for more information.

    Maximum EventsMaximum Size (MB)Timeout (seconds)
    None11