New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Rehydrating from Archives

Log Archive Rehydrating is in public beta. Request early access by contacting Datadog Support. Rehydrating is currently only supported for AWS S3 Archives, and for users of the Datadog US site. Send feedback for this feature.

Overview

Log Rehydration* enables you to capture log events from customer-owned storage-optimized archives back into Datadog’s search-optimized Log Explorer, so that you can use Datadog to analyze or investigate log events that are either old or were excluded from indexing.

Historical views

With historical views, teams rehydrate archived log events precisely by timeframe and query filter to meet specific, unexpected use cases efficiently. To create a historical view, go the Configuration page of your Datadog account and select the “Rehydrate From Archives” tab, then the “New Historical View” button.

Add new historical views

  1. Select the archive from which you wish to rehydrate log events. Only archives that are configured to use role delegation are available for rehydrating.

  2. Choose the time period for which you wish to rehydrate log events. The time period must be older than 24 hours.

  3. Input the query. The query syntax is the same as that of the log explorer search, but is limited to log attributes, reserved attributes, and free text search on the message. You must include a service argument.

  4. Name your historical view. Names must begin with a lowercase letter and can only contain lowercase letters, numbers, and the - character.

  5. (Optional) Add a description to give your team context about the purpose of the historical view.

A historical view can contain a maximum of 300 million log events. There is no limit to how large its time range can be, but if you expect a historical view may exceed that limit, make your query filter more specific.

View historical view content

From the historical view page

After selecting “Rehydrate from Archive,” the historical view is marked as “pending” until its content is ready to be queried.

Once the content is rehydrated, the historical view is marked as active, and the link in the query column leads to the historical view in the log explorer.

From the log explorer

Alternatively, teams can find the historical view from the Log Explorer directly from the index selector. When selecting a historical view, a pop-up offers to set the timeframe to one that is relevant to the selected historical view.

Deleting historical views

Historical views stay in Datadog until you opt to delete them. You can mark a historical view to be deleted by selecting and confirming the delete icon at the far right of the historical view.

24 hours later, the historical view is definitively deleted; until that time, the team is able to cancel the deletion.

Setting up archive rehydrating

Define a Datadog archive

An external archive must be configured in order to rehydrate data from it. Follow the guide to archive your logs in the available destinations.

Permissions

Datadog requires the permission to read from your archives in order to rehydrate content from them. This permission can be changed at any time.

AWS S3 Bucket

In order to rehydrate log events from your archives, Datadog uses the IAM Role in your AWS account that you configured for your AWS integration. If you have not yet created that Role, follow these steps to do so. To allow that Role to rehydrate log events from your archives, add the following permission statement to its IAM policies. Be sure to edit the bucket names and, if desired, specify the paths that contain your log archives.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DatadogUploadAndRehydrateLogArchives",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*",
                "arn:aws:s3:::<MY_BUCKET_NAME_2_/_MY_OPTIONAL_BUCKET_PATH_2>/*"
            ]
        },
        {
            "Sid": "DatadogRehydrateLogArchivesListBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::<MY_BUCKET_NAME_1>",
                "arn:aws:s3:::<MY_BUCKET_NAME_2>"
            ]
        }
    ]
}

Adding role delegation to S3 archives

Datadog only supports rehydrating from archives that have been configured to use role delegation to grant access. Once you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your archive configuration page has the correct AWS Account + Role combination.

Feedback

Datadog would love to receive your feedback on this feature, especially as it is in early beta. You can submit your feedback directly from this form.

*Log Rehydration is a trademark of Datadog, Inc.