Overview

Archive Search lets you query logs directly from long-term object storage archives, without indexing them in advance. Use Archive Search for immediate access to archived logs, for investigations, audits, or troubleshooting beyond your indexing retention period.

Archive Search differs from Rehydration by streaming results in real time as data is scanned, rather than running as a background batch job. It’s more cost-effective, charging only for the scan itself with the first 100,000 logs retained temporarily at no cost, and faster.

When you launch a search:

• Logs stream into a dedicated results page.
• Up to 100,000 logs are retained for 24 hours.
• You can optionally index results before or after the search to keep them longer and make them available throughout Datadog.

This feature supports logs archived through:

• Datadog Log Management archives
• Observability Pipelines archives

Typical use cases

Archive Search is ideal when you need to query logs that are stored in an external archive. Common use cases include:

• Incident investigations: Retrieve logs tied to a transaction_id, user_id, or session_id that fall outside your indexing retention.
  Example: Query logs from three weeks ago using a specific user_id, even if your indexed retention is only 15 days.

• Security analysis: Examine archived logs to investigate login attempts or other activity by IP or user.
  Example: Retrieve all login attempts from a specific IP address across the last 12 months.

• Compliance and audit support: Access archived customer or billing logs for audits without permanently re-indexing large volumes of data.
  Example: Query invoice-related logs (customer_id:12345, service:billing) from the past 18 months for a tax audit.

Prerequisites

Before using Archive Search:

1. Configure an external archive (Amazon S3, Azure Storage, or Google Cloud Storage). See Log Archives.
2. Ensure Datadog has permission to read from the archive, see Cloud-specific permissions.
   • Amazon S3: IAM role delegation
   • Azure Storage: Azure AD with Storage Blob Data Contributor role
   • Google Cloud Storage: Service account with Storage Object Viewer role

Permissions

Archive Search results are visible to all users in your organization who have access to the Archive Search feature. However, restriction queries, such as log security filters and data restrictions configured in Datadog, are still enforced on the result page and apply to all users. This means each user may only see logs they are authorized to view based on organization-wide permissions and filters.

For more information on access controls and log security, see How to Set Up RBAC for Logs.

Launching a search

1. Go to Logs > Archive Search > New Search.
2. Select an Archive and time range.
3. Enter a query, such as user_id:abc123.
4. (Optional) Rename the search.
5. (Optional) Enable indexing before launching the search.
6. Click Search.

Logs stream into the results page in real time. A progress bar shows scan status, and you can stop the search at any time.

Query preview

When you create or configure a search, Datadog downloads a small sample (up to 1,000 logs) from the selected archive and time range. Use this preview to verify query syntax, inspect log structure, and adjust filters.

Note: The preview sample may not include logs that match your query. It is for validation and exploration only.

View and retain results

By default, you are charged only for the scan. The first 100,000 logs are stored temporarily (24 hours) at no cost and accessible directly in Archive Search results pages, where you can click on any log to see its full details and context. After 24 hours, results expire automatically.

To retain more data or access logs in other Datadog products, choose one of the following:

• Index before launch: Retain more than 100,000 logs, set a custom retention period (for example, 7, 15, or 30 days), and access results across the platform immediately.
• Index after completion: During the 24-hour window, you can index results to extend retention and make them available in Log Explorer, Dashboards, and Notebooks.

Analyze results

After launching a search, logs stream into the Archive Search Results page. From this page, you can use filters to narrow down the results and open specific log details to investigate issues.

Limitations

While Archive Search provides access to archived logs, it has limited analytical capabilities compared to indexed logs:

• No aggregations or analytics: You cannot run aggregations, create visualizations, or perform advanced analytics directly on Archive Search results.
• Results page only: Archive Search results are only available in the dedicated results page and cannot be queried from other parts of the Datadog platform (such as Dashboards, Notebooks, or Log Explorer).

To enable full analytics and platform-wide visibility, you need to index the search results (either before launching the search or after completion within the 24-hour window). When indexed, your logs become available across all Datadog products with full aggregation, visualization, and analytics capabilities.

Manage searches

From the Archive Search list view, you can:

• Stop a running search: preserves logs already retrieved.
• Duplicate a search: opens the Archive Search creation form with the same parameters for efficient reruns.

Search performance and scan volume

Archive Search scans archived log files within the selected time range. Scan volume is the total size of those files read during the query. Large scan volumes can increase search time and cost.

To improve query performance and reduce scan volume:

• Narrow the time range and use selective filters.
• Administrators with Logs Write Archives permission can set maximum log limits and available retention durations.

Cloud-specific permissions

Datadog requires the permission to read your archives to search content from them. This permission can be changed at any time.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DatadogUploadAndRehydrateLogArchives",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": [
        "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*",
        "arn:aws:s3:::<MY_BUCKET_NAME_2_/_MY_OPTIONAL_BUCKET_PATH_2>/*"
      ]
    },
    {
      "Sid": "DatadogRehydrateLogArchivesListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": [
        "arn:aws:s3:::<MY_BUCKET_NAME_1>",
        "arn:aws:s3:::<MY_BUCKET_NAME_2>"
      ]
    }
  ]
}

Further reading

Additional helpful documentation, links, and articles:

Explore logs in DatadogDOCUMENTATION Configure Log ArchivesDOCUMENTATION Manage log retention and indexingDOCUMENTATION