Network Performance Monitoring is now generally available! Network Monitoring is now available!

Rehydrating from Archives

Overview

Log Rehydration* enables you to capture log events from customer-owned storage-optimized archives back into Datadog’s search-optimized Log Explorer, so that you can use Datadog to analyze or investigate log events that are either old or were excluded from indexing.

Historical views

With historical views, teams rehydrate archived log events precisely by timeframe and query filter to meet specific, unexpected use cases efficiently. To create a historical view, go the Configuration page of your Datadog account and select the “Rehydrate From Archives” tab, then the “New Historical View” button.

Add new historical views

  1. Select the archive from which you wish to rehydrate log events. Only archives that are configured to use role delegation are available for rehydrating.

  2. Choose the time period for which you wish to rehydrate log events. The time period must be older than 24 hours.

  3. Input the query. The query syntax is the same as that of the log explorer search, but is limited to log attributes, reserved attributes, and free text search on the message.

  4. Name your historical view. Names must begin with a lowercase letter and can only contain lowercase letters, numbers, and the - character.

  5. (Optional) Add a description to give your team context about the purpose of the historical view.

A historical view can contain a maximum of 300 million log events. There is no limit to how large its time range can be, but if you expect a historical view may exceed that limit, make your query filter more specific.

View historical view content

From the historical view page

After selecting “Rehydrate from Archive,” the historical view is marked as “pending” until its content is ready to be queried.

Once the content is rehydrated, the historical view is marked as active, and the link in the query column leads to the historical view in the log explorer.

From the log explorer

Alternatively, teams can find the historical view from the Log Explorer directly from the index selector. When selecting a historical view, a pop-up offers to set the timeframe to one that is relevant to the selected historical view.

Deleting historical views

Historical views stay in Datadog until you opt to delete them. You can mark a historical view to be deleted by selecting and confirming the delete icon at the far right of the historical view.

24 hours later, the historical view is definitively deleted; until that time, the team is able to cancel the deletion.

Setting up archive rehydrating

Define a Datadog archive

An external archive must be configured in order to rehydrate data from it. Follow the guide to archive your logs in the available destinations.

Permissions

Datadog requires the permission to read from your archives in order to rehydrate content from them. This permission can be changed at any time.

In order to rehydrate log events from your archives, Datadog uses the IAM Role in your AWS account that you configured for your AWS integration. If you have not yet created that Role, follow these steps to do so. To allow that Role to rehydrate log events from your archives, add the following permission statement to its IAM policies. Be sure to edit the bucket names and, if desired, specify the paths that contain your log archives.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DatadogUploadAndRehydrateLogArchives",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*",
                "arn:aws:s3:::<MY_BUCKET_NAME_2_/_MY_OPTIONAL_BUCKET_PATH_2>/*"
            ]
        },
        {
            "Sid": "DatadogRehydrateLogArchivesListBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::<MY_BUCKET_NAME_1>",
                "arn:aws:s3:::<MY_BUCKET_NAME_2>"
            ]
        }
    ]
}

Adding role delegation to S3 archives

Datadog only supports rehydrating from archives that have been configured to use role delegation to grant access. Once you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your archive configuration page has the correct AWS Account + Role combination.

In order to rehydrate log events from your archives, Datadog uses a service account with the Storage Object Viewer role. You can grant this role to your Datadog service account from the GCP IAM Admin page by editing the service account’s permissions, adding another role, and then selecting Storage > Storage Object Viewer.

*Log Rehydration is a trademark of Datadog, Inc.