Overview

Use the Amazon S3 destination to send logs in JSON or Parquet format to Amazon S3. See Automatically generated Parquet schema.

You can also route logs to Snowflake using the Amazon S3 destination.

Note: If you want to send logs to an S3 bucket, and later be able to rehydrate them for analysis and investigation in Datadog, use the Datadog Archives destination.

Set up an Amazon S3 bucket

Create an Amazon S3 bucket

1. Navigate to Amazon S3 buckets.
2. Click Create bucket.
3. Enter a descriptive name for your bucket.
4. Do not make your bucket publicly readable.
5. Optionally, add tags.
6. Click Create bucket.

Set up an IAM policy that allows Workers to write to the S3 bucket

1. Navigate to the IAM console.
2. Select Policies in the left side menu.
3. Click Create policy.
4. Click JSON in the Specify permissions section.
5. Copy the below policy and paste it into the Policy editor. Replace <MY_BUCKET_NAME_1>/<MY_OPTIONAL_BUCKET_PATH_1> with the information for the S3 bucket you created in the previous section.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "DatadogOPUpload",
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject"
               ],
               "Resource": "arn:aws:s3:::<MY_BUCKET_NAME_1>/<MY_OPTIONAL_BUCKET_PATH_1>/*"
           }
       ]
   }
   

6. Click Next.
7. Enter a descriptive policy name.
8. Optionally, add tags.
9. Click Create policy.

Set up the destination for your pipeline

Set up the Amazon S3 destination and its environment variables when you create a pipeline. The information below is configured in the pipelines UI.

1. Enter your S3 bucket name. If you configured Log Archives, it’s the name of the bucket you created earlier.
2. Enter the AWS region the S3 bucket is in.
3. (Optional) Enter the key prefix.
   • Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in / to act as a directory path; a trailing / is not automatically added.
     • See template syntax if you want to route logs to different object keys based on specific fields in your logs.
   • Notes:
     • Datadog recommends that you start your prefixes with the directory name and without a lead slash (/). For example, app-logs/ or service-logs/.
     • Do not use the same S3 prefix as a Datadog Archives destination. The Amazon S3 destination writes files in a different format and having both file types in the same prefix can result in rehydration issues.
4. Select the storage class for your S3 bucket in the Storage Class dropdown menu.
5. Select the encoding you want to use in the Encoding dropdown menu (JSON or Parquet).
   • Note: For Parquet, the schema is generated per batch and can vary. See Automatically generated Parquet schema.
6. Select a compression algorithm in the Compression - Algorithm dropdown menu. If you selected:
   • Parquet: Datadog recommends snappy or a low-compression level if you choose zstd.
   • JSON: Datadog recommends gzip.

Optional settings

Batching

1. Enter a maximum batching size and select the unit (MB or GB) in the dropdown menu. If not configured, the default is 100 MB.
2. Enter a batching timeout in seconds. If not configured, the default is 900 seconds.

AWS authentication

Select an AWS authentication option. If you are only using the user or role you created earlier for authentication, do not select Assume role. Select Assume role only if the user or role you created earlier needs to assume a different role to access the AWS resource. The assumed role’s permissions must be explicitly defined.
If you select Assume role:

1. Enter the ARN of the IAM role you want to assume.
   • Note: The user or role you created earlier must have permission to assume this role so that the Worker can authenticate with AWS.
2. (Optional) Enter the assumed role session name and external ID.

Buffering

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Set secrets

These are the defaults used for secret identifiers and environment variables.

Note: If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

Route logs to Snowflake using the Amazon S3 destination

You can route logs from Observability Pipelines to Snowflake using the Amazon S3 destination by configuring Snowpipe in Snowflake to automatically ingest those logs. Snowpipe continuously monitors your S3 bucket for new files and automatically ingests them into your Snowflake tables, ensuring near real-time data availability for analytics or further processing. When logs are collected by Observability Pipelines, they are written to an S3 bucket. To set this up:

1. Set up a pipeline to use Amazon S3 as the log destination. Use the configuration detailed in Set up the destination for your pipeline.
2. Set up Snowpipe in Snowflake. See Automating Snowpipe for Amazon S3 for instructions.

How the destination works

AWS Authentication

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

Permissions

The Observability Pipelines Worker requires these policy permissions to send logs to Amazon S3:

• s3:PutObject

Automatically generated Parquet schema

The Observability Pipelines Worker collects a batch of events, generates a schema for those events, and then flushes the batch to S3. The schema can vary between batches because the schema is based on the current batch of events only.

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.