Amazon S3 Source

Use Observability Pipelines’ Amazon S3 source to receive logs from Amazon S3. Select and set up this source when you set up a pipeline.

Prerequisites

To use Observability Pipelines’ Amazon S3 source, you need to:

  • Configure a SQS queue to receive your S3 bucket notifications, which is required to use the Amazon S3 source.
  • Set up AWS authentication using AWS_PROFILE and AWS_CONFIG FILE environment variables. Observability Pipelines uses the credentials associated with those environment variables to collect logs from Amazon S3. See AWS Authentication for more information.

Set up the source in the pipeline UI

Select and set up this source when you set up a pipeline. The information below is for the source settings in the pipeline UI.

  1. Enter the AWS region.
  2. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required:
    • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
    • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
    • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

AWS Authentication

To use the Amazon S3 source, you need to set up AWS credential files and environment variables. Observability Pipelines uses those credentials to collect logs from Amazon S3. Datadog recommends setting up a specific AWS profile that can be used by Observability Pipelines.

To set up AWS authentication:

  1. Create an IAM role if you don’t have one already. The role needs, at a minimum, these permissions to interact with the component. See Create a role to delegate permissions to an IAM user for more information.
  2. In your AWS configuration file, create a new profile using the role_arn from the role you created in step 1.
  3. When installing the Observability Pipelines Worker, ensure you set the AWS_PROFILE and AWS_CONFIG_FILE environment variables. The AWS_CONFIG_FILE variable is the path to your AWS configuration file. Set AWS_PROFILE to the name of the profile you created in step 2. See Configuration and credential file setting in the AWS CLI for more information. This is an example of a profile configuration:
    [profile profile_name]
region = us-east-1
output = json
role_arn = arn:aws:iam::123456789:role/MyRole
source_profile = default

Permissions

For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

  • s3:GetObject
  • sqs:ReceiveMessage
  • sqs:DeleteMessage