---
isPrivate: true
title: (LEGACY) Route Logs in Datadog-Rehydratable Format to Amazon S3
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: >-
  Docs > Observability Pipelines > (LEGACY) Observability Pipelines
  Documentation > (LEGACY) Observability Pipelines Guides > (LEGACY) Route Logs
  in Datadog-Rehydratable Format to Amazon S3
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# (LEGACY) Route Logs in Datadog-Rehydratable Format to Amazon S3

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

{% alert level="danger" %}
The Observability Pipelines Datadog Archives destination is in beta.
{% /alert %}

## Overview{% #overview %}

The Observability Pipelines `datadog_archives` destination formats logs into a Datadog-rehydratable format and then routes them to [Log Archives](https://docs.datadoghq.com/logs/log_configuration/archives.md). These logs are not ingested into Datadog, but are routed directly to the archive. You can then rehydrate the archive in Datadog when you need to analyze and investigate them.

The Observability Pipelines Datadog Archives destination is useful when:

- You have a high volume of noisy logs, but you may need to index them in Log Management ad hoc.
- You have a retention policy.

For example in this first diagram, some logs are sent to a cloud storage for archiving and others to Datadog for analysis and investigation. However, the logs sent directly to cloud storage cannot be rehydrated in Datadog when you need to investigate them.

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/guide/datadog_archives/op-cloud-storage.cbaece7618c42d9a688f4862be9d69e0.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/guide/datadog_archives/op-cloud-storage.cbaece7618c42d9a688f4862be9d69e0.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="A diagram showing logs going to cloud storage and Datadog." /%}

In this second diagram, all logs are going to the Datadog Agent, including the logs that went to a cloud storage in the first diagram. However, in the second scenario, before the logs are ingested into Datadog, the `datadog_archives` destination formats and routes the logs that would have gone directly to a cloud storage to Datadog Log Archives instead. The logs in Log Archive can be rehydrated in Datadog when needed.

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/guide/datadog_archives/op-datadog-archives.9fc2abf8331987a4acf0a65028f8e68f.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/guide/datadog_archives/op-datadog-archives.9fc2abf8331987a4acf0a65028f8e68f.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="A diagram showing all logs going to Datadog." /%}

This guide walks you through how to:

- Configure a Log Archive
- Configure the `datadog_archives` destination
- Rehydrate your archive

`datadog_archives` is available for Observability Pipelines Worker version 1.5 and later.

## Configure a Log Archive{% #configure-a-log-archive %}

### Create an Amazon S3 bucket{% #create-an-amazon-s3-bucket %}

{% callout %}
# Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, uk1.datadoghq.com

See [AWS Pricing](https://aws.amazon.com/s3/pricing/) for inter-region data transfer fees and how cloud storage costs may be impacted.
{% /callout %}

1. Navigate to [Amazon S3 buckets](https://s3.console.aws.amazon.com/s3/home).
1. Click **Create bucket**.
1. Enter a descriptive name for your bucket.
1. Do not make your bucket publicly readable.
1. Optionally, add tags.
1. Click **Create bucket**.

### Set up an IAM policy that allows Workers to write to the S3 bucket{% #set-up-an-iam-policy-that-allows-workers-to-write-to-the-s3-bucket %}

1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/).
1. Select **Policies** in the left side menu.
1. Click **Create policy**.
1. Click **JSON** in the **Specify permissions** section.
1. Copy the below policy and paste it into the **Policy editor**. Replace `<MY_BUCKET_NAME>` and `<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>` with the information for the S3 bucket you created earlier.
   ```json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "DatadogUploadAndRehydrateLogArchives",
               "Effect": "Allow",
               "Action": ["s3:PutObject", "s3:GetObject"],
               "Resource": "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*"
           },
           {
               "Sid": "DatadogRehydrateLogArchivesListBucket",
               "Effect": "Allow",
               "Action": "s3:ListBucket",
               "Resource": "arn:aws:s3:::<MY_BUCKET_NAME>"
           }
       ]
   }
```
1. Click **Next**.
1. Enter a descriptive policy name.
1. Optionally, add tags.
1. Click **Create policy**.

{% tab title="Docker" %}
### Create an IAM user{% #create-an-iam-user %}

Create an IAM user and attach the IAM policy you created earlier to it.

1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/).
1. Select **Users** in the left side menu.
1. Click **Create user**.
1. Enter a user name.
1. Click **Next**.
1. Select **Attach policies directly**.
1. Choose the IAM policy you created earlier to attach to the new IAM user.
1. Click **Next**.
1. Optionally, add tags.
1. Click **Create user**.

Create access credentials for the new IAM user. Save these credentials as `AWS_ACCESS_KEY` and `AWS_SECRET_ACCESS_KEY`.
{% /tab %}

{% tab title="AWS EKS" %}
### Create a service account{% #create-a-service-account %}

[Create a service account](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) to use the policy you created above. In the Helm configuration, replace `${DD_ARCHIVES_SERVICE_ACCOUNT}` with the name of the service account.
{% /tab %}

{% tab title="APT-based Linux" %}
### Create an IAM user{% #create-an-iam-user %}

Create an IAM user and attach the IAM policy you created earlier to it.

1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/).
1. Select **Users** in the left side menu.
1. Click **Create user**.
1. Enter a user name.
1. Click **Next**.
1. Select **Attach policies directly**.
1. Choose the IAM policy you created earlier to attach to the new IAM user.
1. Click **Next**.
1. Optionally, add tags.
1. Click **Create user**.

Create access credentials for the new IAM user. Save these credentials as `AWS_ACCESS_KEY` and `AWS_SECRET_ACCESS_KEY`.
{% /tab %}

{% tab title="RPM-based Linux" %}
### Create an IAM user{% #create-an-iam-user %}

Create an IAM user and attach the IAM policy you created earlier to it.

1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/).
1. Select **Users** in the left side menu.
1. Click **Create user**.
1. Enter a user name.
1. Click **Next**.
1. Select **Attach policies directly**.
1. Choose the IAM policy you created earlier to attach to the new IAM user.
1. Click **Next**.
1. Optionally, add tags.
1. Click **Create user**.

Create access credentials for the new IAM user. Save these credentials as `AWS_ACCESS_KEY` and `AWS_SECRET_ACCESS_KEY`.
{% /tab %}

{% tab title="Terraform (AWS)" %}
### Attach the policy to the IAM instance profile{% #attach-the-policy-to-the-iam-instance-profile %}

Attach the policy to the IAM Instance Profile that is created with Terraform, which you can find under the `iam-role-name` output.
{% /tab %}

### Connect the S3 bucket to Datadog Log Archives{% #connect-the-s3-bucket-to-datadog-log-archives %}

1. Navigate to Datadog [Log Forwarding](https://app.datadoghq.com/logs/pipelines/log-forwarding).
1. Click **Add a new archive**.
1. Enter a descriptive archive name.
1. Add a query that filters out all logs going through log pipelines so that none of those logs go into this archive. For example, add the query `observability_pipelines_read_only_archive`, assuming no logs going through the pipeline have that tag added.
1. Select **AWS S3**.
1. Select the AWS Account that your bucket is in.
1. Enter the name of the S3 bucket.
1. Optionally, enter a path.
1. Check the confirmation statement.
1. Optionally, add tags and define the maximum scan size for rehydration. See [Advanced settings](https://docs.datadoghq.com/logs/log_configuration/archives.md#advanced-settings) for more information.
1. Click **Save**.

See the [Log Archives documentation](https://docs.datadoghq.com/logs/log_configuration/archives.md) for additional information.

## Configure the `datadog_archives` destination{% #configure-the-datadog_archives-destination %}

You can configure the `datadog_archives` destination using the configuration file or the pipeline builder UI.

{% alert level="danger" %}
If the Worker is ingesting logs that are not coming from the Datadog Agent and are routed to the Datadog Archives destination, those logs are not tagged with [reserved attributes](https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention.md#reserved-attributes). This means that you lose Datadog telemetry and the benefits of [unified service tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md?tab=kubernetes). For example, say your syslogs are sent to `datadog_archives` and those logs have the status tagged as `severity` instead of the reserved attribute of `status` and the host tagged as `hostname` instead of the reserved attribute `host`. When these logs are rehydrated in Datadog, the `status` for the logs are all set to `info` and none of the logs will have a hostname tag.
{% /alert %}

### Configuration file{% #configuration-file %}

For manual deployments, the [sample pipelines configuration file](https://docs.datadoghq.com/observability_pipelines/legacy/setup/datadog_with_archiving.md#install-the-observability-pipelines-worker) for Datadog includes a sink for sending logs to Amazon S3 under a Datadog-rehydratable format.

{% tab title="Docker" %}
In the sample pipelines configuration file, replace `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` with the AWS credentials you created earlier.
{% /tab %}

{% tab title="AWS EKS" %}
In the sample pipelines configuration file, replace `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` with the AWS credentials you created earlier.
{% /tab %}

{% tab title="APT-based Linux" %}
In the sample pipelines configuration file, replace `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` with the AWS credentials you created earlier.
{% /tab %}

{% tab title="RPM-based Linux" %}
In the sample pipelines configuration file, replace `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` with the AWS credentials you created earlier.
{% /tab %}

{% tab title="Terraform (AWS)" %}
Replace `${DD_ARCHIVES_BUCKET}` and $`{DD_ARCHIVES_REGION}` parameters based on your S3 configuration.
{% /tab %}

### Pipeline builder UI{% #pipeline-builder-ui %}

1. Navigate to your [Pipeline](https://app.datadoghq.com/observability-pipelines/).
1. (Optional) Add a remap transform to tag all logs going to `datadog_archives`.
   1. Click **Edit** and then **Add More** in the **Add Transforms.
   1. Click the **Remap** tile.
   1. Enter a descriptive name for the component.
   1. In the **Inputs** field, select the source to connect this destination to.
   1. Add `.sender = "observability_pipelines_worker"` in the **Source** section.
   1. Click **Save**.
   1. Navigate back to your pipeline.
1. Click **Edit**.
1. Click **Add More** in the **Add Destination** tile.
1. Click the **Datadog Archives** tile.
1. Enter a descriptive name for the component.
1. Select the sources or transforms to connect this destination to.

{% tab title="AWS S3" %}
In the **Bucket** field, enter the name of the S3 bucket you created earlier.Enter `aws_s3` in the **Service** field.Toggle **AWS S3** to enable those specific configuration options.In the **Storage Class** field, select the storage class in the dropdown menu.Set the other configuration options based on your use case.Click **Save**.
{% /tab %}

{% tab title="Azure Blob" %}
In the **Bucket** field, enter the name of the S3 bucket you created earlier.Enter `azure_blob` in the **Service** field.Toggle **Azure Blob** to enable those specific configuration options.Enter the Azure Blob Storage Account connection string.Set the other configuration options based on your use case.Click **Save**.
{% /tab %}

{% tab title="GCP Cloud Storage" %}
In the **Bucket** field, enter the name of the S3 bucket you created earlier.Enter `gcp_cloud_storage` in the **Service** field.Toggle **GCP Cloud Storage** to enable those specific configuration options.Set the configuration options based on your use case.Click **Save**.
{% /tab %}

If you are using Remote Configuration, deploy the change to your pipeline in the UI. For manual configuration, download the updated configuration and restart the worker.

See [Datadog Archives reference](https://docs.datadoghq.com/observability_pipelines/legacy/reference/sinks.md#datadogarchivessink) for details on all configuration options.

## Rehydrate your archive{% #rehydrate-your-archive %}

See [Rehydrating from Archives](https://docs.datadoghq.com/logs/log_configuration/rehydrating.md) for instructions on how to rehydrate your archive in Datadog so that you can start analyzing and investigating those logs.

## Further reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Working with data in Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/legacy/working_with_data.md)
- [Learn more about Log Archives](https://docs.datadoghq.com/logs/log_configuration/archives.md)
- [Learn more about rehydrating log archives](https://docs.datadoghq.com/logs/log_configuration/rehydrating.md)