---
title: Amazon Data Firehose Source
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: Docs > Observability Pipelines > Sources > Amazon Data Firehose Source
---

# Amazon Data Firehose Source

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). ().
{% /alert %}

{% /callout %}
Available for:
{% icon name="icon-logs" /%}
 Logs 
## Overview{% #overview %}

Use Observability Pipelines' Amazon Data Firehose source to receive logs from Amazon Data Firehose.

## Prerequisites{% #prerequisites %}

To use Observability Pipelines' Amazon Data Firehose:

- Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See [Understand HTTP endpoint delivery request and response specifications](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#requestformat) for more information.
- If your forwarders are globally configured to enable SSL, you need the appropriate TLS certificates and the password you used to create your private key.

## Setup{% #setup %}

Set up this source when you [set up a pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines/). You can set up a pipeline in the [UI](https://app.datadoghq.com/observability-pipelines), using the [API](https://docs.datadoghq.com/api/latest/observability-pipelines/), or with [Terraform](https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline). The instructions in this section are for setting up the source in the UI.

{% alert level="danger" %}
Only enter the identifiers for the Amazon Data Firehose address and, if applicable, the TLS key pass. Do not enter the actual values.
{% /alert %}

- Enter the identifier for your Amazon Data Firehose address. If you leave it blank, the default is used.

### Optional settings{% #optional-settings %}

#### AWS authentication{% #aws-authentication %}

Select an **AWS authentication** option. If you select **Assume role**:

1. Enter the ARN of the IAM role you want to assume.
1. Optionally, enter the assumed role session name and external ID.

#### Enable TLS{% #enable-tls %}

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/) for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.

- Enter the identifier for your Amazon Data Firehose key pass. If you leave it blank, the default is used.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

## Set secrets{% #set-secrets %}

These are the defaults used for secret identifiers and environment variables.

**Note**: If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with `DD_OP`. For example, if you entered `PASSWORD_1` for a password identifier, the environment variable for that password is `DD_OP_PASSWORD_1`.

{% tab title="Secrets Management" %}

- Amazon Data Firehose address identifier:
  - References the socket address on which the Observability Pipelines Worker listens to receive logs.
  - The default identifier is `SOURCE_AWS_DATA_FIREHOSE_ADDRESS`.
- Amazon Data Firehose TLS passphrase identifier (when TLS is enabled):
  - The default identifier is `SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`.

{% /tab %}

{% tab title="Environment variables" %}

- Amazon Data Firehose address:
  - The Observability Pipelines Worker listens to this socket address to receive logs from Amazon Data Firehose.
  - The default environment variable is `DD_OP_SOURCE_AWS_DATA_FIREHOSE_ADDRESS`.
- Amazon Data Firehose TLS passphrase (when enabled):
  - The default environment variable is `DD_OP_SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`.

{% /tab %}

## Send logs to the Observability Pipelines Worker over Amazon Data Firehose{% #send-logs-to-the-observability-pipelines-worker-over-amazon-data-firehose %}

Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See [Understand HTTP endpoint delivery request and response specifications](https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#requestformat) for more information.

To send logs to the Observability Pipelines Worker, set up an Amazon Data Firehose stream with an [HTTP endpoint destination](https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html?icmpid=docs_console_unmapped#create-destination-http) in the region where your logs are. Configure the endpoint URL to the endpoint where OPW is deployed.

## AWS Authentication{% #aws-authentication-1 %}

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See [AWS SDKs and Tools standardized credential providers](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html) for more information.

### Permissions{% #permissions %}

For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

- `s3:GetObject`
- `sqs:ReceiveMessage`
- `sqs:DeleteMessage`