New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Archives on AWS S3

Overview

Keeping your logs in a storage-optimized archive for longer periods of time is a great way to meet compliance requirements and retain auditability for ad-hoc investigations within budget. Once your logs are being archived for the long-term, you always have access to them should you ever need to investigate something unexpected or that might have happened a long time ago.

This guide shows you how to set up an archive for forwarding ingested logs to your own Amazon S3 bucket.

Create and Configure an S3 Bucket

Go into your AWS Console and create an S3 bucket to send your archives to. Be careful not to make your bucket publicly readable.

Next, grant Datadog permissions to write log archives to your S3 bucket. You can grant those permissions either from your bucket policy or with role delegation.

Note: To rehydrate logs from your archive back into Datadog, the role delegation setup is required.

  1. Modify your bucket to grant write-only access to the Datadog AWS user. Do this by editing your bucket’s permissions, and setting the bucket policy with the following content (replace <MY_BUCKET_NAME> with the name of your bucket):

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowDatadogArchivesUploader",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "464622532012"
                },
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl",
                    "s3:ListMultipartUploadParts",
                    "s3:AbortMultipartUpload"
                ],
                "Resource": "arn:aws:s3:::<MY_BUCKET_NAME>/*"
            }
        ]
    }
    
  1. Go to your Pipelines page in Datadog, and select the Add archive option at the bottom. Only Datadog users with admin status can complete this and the following step.

  2. Input your bucket name. Optionally input a prefix directory for all the content of your log archives. Save your archive, and you are finished.

Use of role delegation for writing log archives is in beta. Contact Datadog support to have it enabled for your account.

  1. Set up the AWS integration for the AWS account that holds your S3 bucket. This involves creating a role that Datadog can use to integrate with AWS Cloudwatch.

  2. Add the following permission statement to the IAM policies of your Datadog role (be sure to edit the bucket names and, if desired, specify the paths that contain your log archives). The GetObject and ListBucket permissions allow for Rehydrating from Archives. The PutObject permissions is sufficient for uploading archives.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DatadogUploadAndRehydrateLogArchives",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*",
                "arn:aws:s3:::<MY_BUCKET_NAME_2_/_MY_OPTIONAL_BUCKET_PATH_2>/*"
            ]
        },
        {
            "Sid": "DatadogRehydrateLogArchivesListBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::<MY_BUCKET_NAME_1>",
                "arn:aws:s3:::<MY_BUCKET_NAME_2>"
            ]
        }
    ]
}
  1. Go to your Pipelines page in Datadog, and select the Add archive option at the bottom. Only Datadog users with admin status can complete this and the following step.

  2. Select the appropriate AWS account + role combination for your S3 bucket. Input your bucket name. Optionally input a prefix directory for all the content of your log archives. Save your archive, and you are finished.

From the moment that your archive settings are successfully configured in your Datadog account, all the logs that Datadog ingests are enriched by your processing Pipelines and subsequently forwarded to your S3 bucket for archiving.

However, after creating or updating your archive configurations, it can take several minutes before the next archive upload is attempted, so you should check back on your S3 bucket in 15 minutes to make sure the archives have successfully been getting uploaded from your Datadog account.

Format of the S3 Archives

The log archives that Datadog forwards to your S3 bucket are in zipped (gzip) JSON format. Under whatever prefix you indicate (or / if there is none), the archives are stored in a directory structure that indicates on what date and at what time the archive files were generated, like so:

/my/s3/prefix/dt=20180515/hour=14/archive_143201.1234.7dq1a9mnSya3bFotoErfxl.json.gz

This directory structure simplifies the process of querying your historical log archives based on their date.

Within the zipped JSON file, each event’s content is formatted as follows:

{
    "_id": "123456789abcdefg",
    "date": "2018-05-15T14:31:16.003Z",
    "host": "i-12345abced6789efg",
    "source": "source_name",
    "service": "service_name",
    "status": "status_level",
    "message": " ... log message content ... ",
    "attributes": { ... log attributes content ... }
}

Note: Archives only include log content which consists of the message, custom attributes, and reserved attributes of your log events. The log tags (metadata that connects your log data to related metrics and traces) are not included.

Multiple S3 Archives

Admins can route specific logs to an archive by adding a query in the archive’s filter field. Logs enter the first archive whose filter they match on, so it is important to order your archives carefully.

For example, if you create a first archive filtered to the env:prod tag and a second archive without any filter (the equivalent of *), all your production logs would go to one S3 bucket/path and the rest would go to the other.

Server Side Encryption (SSE)

To add server side encryption to your S3 log archives, go to the Properties tab in your S3 bucket and select Default Encryption. Select the AES-256 option and Save.