---
title: Amazon Data Firehose Source
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: Docs > Observability Pipelines > Sources > Amazon Data Firehose Source
---

# Amazon Data Firehose Source

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ().
{% /alert %}

{% /callout %}
Available for:
{% icon name="icon-logs" /%}
 Logs 
## Overview{% #overview %}

Use Observability Pipelines' Amazon Data Firehose source to receive logs from Amazon Data Firehose.

## Prerequisites{% #prerequisites %}

To use Observability Pipelines' Amazon Data Firehose:

- Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See [Understand HTTP endpoint delivery request and response specifications](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#requestformat) for more information.
- If your forwarders are globally configured to enable SSL, you need the appropriate TLS certificates and the password you used to create your private key.

## Setup{% #setup %}

Set up this source when you [set up a pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md). You can set up a pipeline in the [UI](https://app.datadoghq.com/observability-pipelines), using the [API](https://docs.datadoghq.com/api/latest/observability-pipelines.md), or with [Terraform](https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline). The instructions in this section are for setting up the source in the UI.

{% alert level="danger" %}
Only enter the identifiers for the Amazon Data Firehose address and, if applicable, the TLS key pass. Do not enter the actual values.
{% /alert %}

- Enter the identifier for your Amazon Data Firehose address. If you leave it blank, the default is used.

### Optional settings{% #optional-settings %}

#### AWS authentication{% #aws-authentication %}

Select an **AWS authentication** option. If you select **Assume role**:

1. Enter the ARN of the IAM role you want to assume.
1. Optionally, enter the assumed role session name and external ID.

#### Enable TLS{% #enable-tls %}

Toggle the switch to **Enable TLS**.

- If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
- The following certificate and key files are required:
  - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  - **Notes**:
    - The configuration data directory `/var/lib/observability-pipelines-worker/config/` is automatically appended to the file paths. See [Advanced Worker Configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md) for more information.
    - The file must be readable by the `observability-pipelines-worker` group and user.

## Set secrets{% #set-secrets %}

These are the defaults used for secret identifiers and environment variables.

**Note**: If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with `DD_OP`. For example, if you entered `PASSWORD_1` for a password identifier, the environment variable for that password is `DD_OP_PASSWORD_1`.

{% tab title="Secrets Management" %}

- Amazon Data Firehose address identifier:
  - References the socket address on which the Observability Pipelines Worker listens to receive logs.
  - The default identifier is `SOURCE_AWS_DATA_FIREHOSE_ADDRESS`.
- Amazon Data Firehose TLS passphrase identifier (when TLS is enabled):
  - The default identifier is `SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`.

{% /tab %}

{% tab title="Environment variables" %}

- Amazon Data Firehose address:
  - The Observability Pipelines Worker listens to this socket address to receive logs from Amazon Data Firehose.
  - The default environment variable is `DD_OP_SOURCE_AWS_DATA_FIREHOSE_ADDRESS`.
- Amazon Data Firehose TLS passphrase (when enabled):
  - The default environment variable is `DD_OP_SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`.

{% /tab %}

## Send logs to the Observability Pipelines Worker over Amazon Data Firehose{% #send-logs-to-the-observability-pipelines-worker-over-amazon-data-firehose %}

Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See [Understand HTTP endpoint delivery request and response specifications](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#requestformat) for more information.

To send logs to the Observability Pipelines Worker, set up an Amazon Data Firehose stream with an [HTTP endpoint destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html?icmpid=docs_console_unmapped#create-destination-http) in the region where your logs are. Configure the endpoint URL to the endpoint where OPW is deployed.

Amazon Data Firehose may send log events nested in an array, such as the structure shown below. To extract the records as individual events, use the [Split Array processor](https://docs.datadoghq.com/observability_pipelines/processors/split_array.md) and target the array. For example, `logEvents`. The Split Array processor enables you to break log data from arrays into individual events, making it easier for users to filter, query, and visualize data that was previously buried in layers.

```json
{
    "logEvents": [
        {log 1},
        {log 2},
        {log 3},
        {log n}
  ]
}
```

## AWS Authentication{% #aws-authentication-1 %}

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See [AWS SDKs and Tools standardized credential providers](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html) for more information.

### Permissions{% #permissions %}

For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

- `s3:GetObject`
- `sqs:ReceiveMessage`
- `sqs:DeleteMessage`