Overview

Configure your Datadog account to forward all the logs ingested—whether indexed or not—to a cloud storage system of your own. Keep your logs in a storage-optimized archive for longer periods of time and meet compliance requirements while also keeping auditability for ad-hoc investigations, with Rehydration.

Navigate to the Log Forwarding page to set up an archive for forwarding ingested logs to your own cloud-hosted storage bucket.

1. If you haven’t already, set up a Datadog integration for your cloud provider.
2. Create a storage bucket.
3. Set permissions to read and/or write on that archive.
4. Route your logs to and from that archive.
5. Configure advanced settings such as encryption, storage class, and tags.
6. Validate your setup and check for possible misconfigurations that Datadog would be able to detect for you.

See how to archive your logs with Observability Pipelines if you want to route your logs to a storage-optimized archive directly from your environment.

Configure an archive

Set up an integration

Create a storage bucket

Set permissions

Only Datadog users with the logs_write_archive permission can create, modify, or delete log archive configurations.

1. Create a policy with the following permission statements:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "DatadogUploadAndRehydrateLogArchives",
         "Effect": "Allow",
         "Action": ["s3:PutObject", "s3:GetObject"],
         "Resource": [
           "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*",
           "arn:aws:s3:::<MY_BUCKET_NAME_2_/_MY_OPTIONAL_BUCKET_PATH_2>/*"
         ]
       },
       {
         "Sid": "DatadogRehydrateLogArchivesListBucket",
         "Effect": "Allow",
         "Action": "s3:ListBucket",
         "Resource": [
           "arn:aws:s3:::<MY_BUCKET_NAME_1>",
           "arn:aws:s3:::<MY_BUCKET_NAME_2>"
         ]
       }
     ]
   }
   

   • The GetObject and ListBucket permissions allow for rehydrating from archives.
   • The PutObject permission is sufficient for uploading archives.
   • Ensure that the resource value under the s3:PutObject and s3:GetObject actions ends with /* because these permissions are applied to objects within the buckets.

2. Edit the bucket names.

3. Optionally, specify the paths that contain your log archives.

4. Attach the new policy to the Datadog integration role.

   • Navigate to Roles in the AWS IAM console.
   • Locate the role used by the Datadog integration. By default it is named DatadogIntegrationRole, but the name may vary if your organization has renamed it. Click the role name to open the role summary page.
   • Click Add permissions, and then Attach policies.
   • Enter the name of the policy created above.
   • Click Attach policies.

1. Grant the Datadog app permission to write to and rehydrate from your storage account.
2. Select your storage account from the Storage Accounts page, go to Access Control (IAM), and select Add -> Add Role Assignment.
3. Input the Role called Storage Blob Data Contributor, select the Datadog app which you created to integrate with Azure, and save.

1. Grant your Datadog Google Cloud service account permissions to write your archives to your bucket.

2. Select your Datadog Google Cloud service account principal from the Google Cloud IAM Admin page and select Edit principal.

3. Click ADD ANOTHER ROLE, select the Storage Object Admin role, and save.

Route your logs to a bucket

Navigate to the Log Forwarding page and select Add a new archive on the Archives tab.

Notes:

• Only Datadog users with the logs_write_archive permission can complete this and the following step.
• Archiving logs to Azure Blob Storage requires an App Registration. See instructions on the Azure integration page, and set the “site” on the right-hand side of the documentation page to “US.” App Registration(s) created for archiving purposes only need the “Storage Blob Data Contributor” role. If your storage bucket is in a subscription being monitored through a Datadog Resource, a warning is displayed about the App Registration being redundant. You can ignore this warning.
• If your bucket restricts network access to specified IPs, add the webhook IPs from the IP ranges list to the allowlist.
• For the US1-FED site, you can configure Datadog to send logs to a destination outside the Datadog GovCloud environment. Datadog is not responsible for any logs that leave the Datadog GovCloud environment. Additionally, Datatdog is not responsible for any obligations or requirements you might have concerning FedRAMP, DoD Impact Levels, ITAR, export compliance, data residency, or similar regulations applicable to these logs after they leave the GovCloud environment.

Advanced settings

Datadog tags

Use this optional configuration step to:

• Include all log tags in your archives (activated by default on all new archives). Note: This increases the size of resulting archives.
• Add tags on rehydrated logs according to your Restriction Queries policy. See the logs_read_data permission.

Define maximum scan size

Use this optional configuration step to define the maximum volume of log data (in GB) that can be scanned for Rehydration on your Log Archives.

For Archives with a maximum scan size defined, all users need to estimate the scan size before they are allowed to start a Rehydration. If the estimated scan size is greater than what is permitted for that Archive, users must reduce the time range over which they are requesting the Rehydration. Reducing the time range will reduce the scan size and allow the user to start a Rehydration.

Storage class

Server side encryption (SSE)

{
    "Id": "key-consolepolicy-3",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<MY_AWS_ACCOUNT_NUMBER>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<MY_AWS_ACCOUNT_NUMBER>:role/<MY_DATADOG_IAM_ROLE_NAME>"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<MY_AWS_ACCOUNT_NUMBER>:role/<MY_DATADOG_IAM_ROLE_NAME>"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

Validation

Once your archive settings are successfully configured in your Datadog account, your processing pipelines begin to enrich all logs ingested into Datadog. These logs are subsequently forwarded to your archive.

However, after creating or updating your archive configurations, it can take several minutes before the next archive upload is attempted. The frequency at which archives are uploaded can vary. Check back on your storage bucket in 15 minutes to make sure the archives are successfully being uploaded from your Datadog account.

After that, if the archive is still in a pending state, check your inclusion filters to make sure the query is valid and matches log events in Live Tail. When Datadog fails to upload logs to an external archive, due to unintentional changes in settings or permissions, the corresponding Log Archive is highlighted in the configuration page.

Hover over the archive to view the error details and the actions to take to resolve the issue. An event is also generated in the Events Explorer. You can create a monitor for these events to detect and remediate failures quickly.

Multiple archives

If multiple archives are defined, logs enter the first archive based on filter.

It is important to order your archives carefully. For example, if you create a first archive filtered to the env:prod tag and a second archive without any filter (the equivalent of *), all your production logs would go to one storage bucket or path, and the rest would go to the other.

Format of the archives

The log archives that Datadog forwards to your storage bucket are in compressed JSON format (.json.gz). Using the prefix you indicate (or / if there is none), the archives are stored in a directory structure that indicates on what date and at what time the archive files were generated, such as the following:

/my/bucket/prefix/dt=20180515/hour=14/archive_143201.1234.7dq1a9mnSya3bFotoErfxl.json.gz
/my/bucket/prefix/dt=<YYYYMMDD>/hour=<HH>/archive_<HHmmss.SSSS>.<DATADOG_ID>.json.gz

This directory structure simplifies the process of querying your historical log archives based on their date.

Within the zipped JSON file, each event’s content is formatted as follows:

{
    "_id": "123456789abcdefg",
    "date": "2018-05-15T14:31:16.003Z",
    "host": "i-12345abced6789efg",
    "source": "source_name",
    "service": "service_name",
    "status": "status_level",
    "message": "2018-05-15T14:31:16.003Z INFO rid='acb-123' status=403 method=PUT",
    "attributes": { "rid": "abc-123", "http": { "status_code": 403, "method": "PUT" } },
    "tags": [ "env:prod", "team:acme" ]
}

Further Reading

Additional helpful documentation, links, and articles:

Learn how to access your archived log content in DatadogDOCUMENTATION Learn about the Log ExplorerDOCUMENTATION Learn about Logging without Limits*DOCUMENTATION