--- title: Log Archives description: Forward all your ingested logs to long term storage. breadcrumbs: Docs > Log Management > Log Configuration > Log Archives --- # Log Archives ## Overview{% #overview %} Configure your Datadog account to forward all the logs ingested—whether [indexed](https://docs.datadoghq.com/logs/indexes/#exclusion-filters) or not—to a cloud storage system of your own. Keep your logs in a storage-optimized archive for longer periods of time and meet compliance requirements while also keeping auditability for ad-hoc investigations, with [Rehydration](https://docs.datadoghq.com/logs/archives/rehydrating/) or [Archive Search](https://docs.datadoghq.com/logs/log_configuration/archive_search/?tab=amazons3). {% image source="https://datadog-docs.imgix.net/images/logs/archives/log_forwarding_archives_122024.2f1c5e847c2f9885a8cc855a2b1715c6.png?auto=format" alt="Archives tab on the Log Forwarding page" /%} Navigate to the [**Log Archiving & Forwarding** page](https://app.datadoghq.com/logs/pipelines/log-forwarding) to set up an archive for forwarding ingested logs to your own cloud-hosted storage bucket. 1. If you haven't already, set up a Datadog integration for your cloud provider. 1. Create a storage bucket. 1. Set permissions to `read` and/or `write` on that archive. 1. Route your logs to and from that archive. 1. Configure advanced settings such as encryption, storage class, and tags. 1. Validate your setup and check for possible misconfigurations that Datadog would be able to detect for you. See how to [archive your logs with Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/explore_templates/?tab=logs#archive-logs) if you want to route your logs to a storage-optimized archive directly from your environment. The following metrics report on logs that have been archived successfully, including logs that were sent successfully after retries. - datadog.archives.logs.bytes - datadog.archives.logs.count ## Configure an archive{% #configure-an-archive %} ### Set up an integration{% #set-up-an-integration %} {% tab title="AWS S3" %} If not already configured, set up the [AWS integration](https://docs.datadoghq.com/integrations/amazon_web_services/?tab=automaticcloudformation#setup) for the AWS account that holds your S3 bucket. - In the general case, this involves creating a role that Datadog can use to integrate with AWS S3. - Specifically for AWS China accounts, use access keys as an alternative to role delegation. {% /tab %} {% tab title="Azure Storage" %} Set up the [Azure integration](https://app.datadoghq.com/account/settings#integrations/azure) within the subscription that holds your new storage account, if you haven't already. This involves [creating an app registration that Datadog can use](https://docs.datadoghq.com/integrations/azure/?tab=azurecliv20#integrating-through-the-azure-portal) to integrate with. **Note:** Archiving to Azure ChinaCloud and Azure GermanyCloud is not supported. Archiving to Azure GovCloud is supported in Preview. To request access, contact Datadog support. {% /tab %} {% tab title="Google Cloud Storage" %} Set up the [Google Cloud integration](https://app.datadoghq.com/account/settings#integrations/google-cloud-platform) for the project that holds your GCS storage bucket, if you haven't already. This involves [creating a Google Cloud service account that Datadog can use](https://docs.datadoghq.com/integrations/google_cloud_platform/?tab=datadogussite#setup) to integrate with. {% /tab %} ### Create a storage bucket{% #create-a-storage-bucket %} {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} Sending logs to an archive is outside of the Datadog GovCloud environment, which is outside the control of Datadog. Datadog shall not be responsible for any logs that have left the Datadog GovCloud environment, including without limitation, any obligations or requirements that the user may have related to FedRAMP, DoD Impact Levels, ITAR, export compliance, data residency or similar regulations applicable to such logs. {% /alert %} {% /callout %} {% tab title="AWS S3" %} Go into your [AWS console](https://s3.console.aws.amazon.com/s3) and [create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html) to send your archives to. {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} Datadog Archives do not support bucket names with dots (.) when integrated with an S3 FIPS endpoint which relies on virtual-host style addressing. Learn more from AWS documentation. [AWS FIPS](https://aws.amazon.com/compliance/fips/) and [AWS Virtual Hosting](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html). {% /alert %} {% /callout %} **Notes:** - Do not make your bucket publicly readable. - For [US1, US3, and US5 sites](https://docs.datadoghq.com/getting_started/site/), see [AWS Pricing](https://aws.amazon.com/s3/pricing/) for inter-region data transfer fees and how cloud storage costs may be impacted. Consider creating your storage bucket in `us-east-1` to manage your inter-region data transfer fees. {% /tab %} {% tab title="Azure Storage" %} - Go to your [Azure Portal](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) and [create a storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal) to send your archives to. Give your storage account a name, select either standard performance or **Block blobs** premium account type, and select the **hot** or **cool** access tier. - Create a **container** service into that storage account. Take note of the container name as you will need to add this in the Datadog Archive Page. **Note:** Do not set [immutability policies](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-immutability-policies-manage) because the last data needs to be rewritten in some rare cases (typically a timeout). {% /tab %} {% tab title="Google Cloud Storage" %} Go to your [Google Cloud account](https://console.cloud.google.com/storage) and [create a GCS bucket](https://cloud.google.com/storage/docs/quickstart-console) to send your archives to. Under **Choose how to control access to objects**, select **Set object-level and bucket-level permissions.** **Note:** Do not add [retention policy](https://cloud.google.com/storage/docs/bucket-lock) because the last data needs to be rewritten in some rare cases (typically a timeout). {% /tab %} ### Set permissions{% #set-permissions %} Only Datadog users with the [`logs_write_archive` permission](https://docs.datadoghq.com/account_management/rbac/permissions/?tab=ui#logs_write_archives) can create, modify, or delete log archive configurations. {% tab title="AWS S3" %} 1. [Create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html) with the following permission statements: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": [ "arn:aws:s3:::/*", "arn:aws:s3:::/*" ] }, { "Sid": "DatadogRehydrateLogArchivesListBucket", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::" ] } ] } ``` - The `GetObject` and `ListBucket` permissions allow for [rehydrating from archives](https://docs.datadoghq.com/logs/archives/rehydrating/). - The `PutObject` permission is sufficient for uploading archives. - Ensure that the resource value under the `s3:PutObject` and `s3:GetObject` actions ends with `/*` because these permissions are applied to objects within the buckets. 1. Edit the bucket names. 1. Optionally, specify the paths that contain your log archives. 1. Attach the new policy to the Datadog integration role. - Navigate to **Roles** in the AWS IAM console. - Locate the role used by the Datadog integration. By default it is named **DatadogIntegrationRole**, but the name may vary if your organization has renamed it. Click the role name to open the role summary page. - Click **Add permissions**, and then **Attach policies**. - Enter the name of the policy created above. - Click **Attach policies**. {% /tab %} {% tab title="Azure Storage" %} 1. Grant the Datadog app permission to write to and rehydrate from your storage account. 1. Select your storage account from the [Storage Accounts page](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts), go to **Access Control (IAM)**, and select **Add -> Add Role Assignment**. 1. Input the Role called **Storage Blob Data Contributor**, select the Datadog app which you created to integrate with Azure, and save. {% image source="https://datadog-docs.imgix.net/images/logs/archives/logs_azure_archive_permissions.4b0233af22b54e0ca976fba47766ae2d.png?auto=format" alt="Add the Storage Blob Data Contributor role to your Datadog App." /%} {% /tab %} {% tab title="Google Cloud Storage" %} 1. Grant your Datadog Google Cloud service account permissions to write your archives to your bucket. 1. Select your Datadog Google Cloud service account principal from the [Google Cloud IAM Admin page](https://console.cloud.google.com/iam-admin/iam) and select **Edit principal**. 1. Click **ADD ANOTHER ROLE**, select the **Storage Object Admin** role, and save. {% image source="https://datadog-docs.imgix.net/images/logs/archives/gcp_role_storage_object_admin-2.3c17c97218c4659171ecc42b3f331a9e.png?auto=format" alt="Add the Storage Object Admin role to your Datadog Google Cloud Service Account." /%} {% /tab %} ### Route your logs to a bucket{% #route-your-logs-to-a-bucket %} Navigate to the [Log Archiving & Forwarding page](https://app.datadoghq.com/logs/pipelines/archives) and select **Add a new archive** on the **Archives** tab. **Notes:** - Only Datadog users with the [`logs_write_archive` permission](https://docs.datadoghq.com/account_management/rbac/permissions/?tab=ui#logs_write_archives) can complete this and the following step. - Archiving logs to Azure Blob Storage requires an App Registration. See instructions [on the Azure integration page](https://docs.datadoghq.com/integrations/azure/), and set the "site" on the right-hand side of the documentation page to "US." App Registration(s) created for archiving purposes only need the "Storage Blob Data Contributor" role. If your storage bucket is in a subscription being monitored through a Datadog Resource, a warning is displayed about the App Registration being redundant. You can ignore this warning. - If your bucket restricts network access to specified IPs, add the webhook IPs from the IP ranges list to the allowlist. - For the **US1-FED site**, you can configure Datadog to send logs to a destination outside the Datadog GovCloud environment. Datadog is not responsible for any logs that leave the Datadog GovCloud environment. Additionally, Datadog is not responsible for any obligations or requirements you might have concerning FedRAMP, DoD Impact Levels, ITAR, export compliance, data residency, or similar regulations applicable to these logs after they leave the GovCloud environment. | Service | Steps | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Amazon S3** | - Select the appropriate AWS account and role combination for your S3 bucket.- Input your bucket name.**Optional**: Input a prefix directory for all the content of your log archives. | | **Azure Storage** | - Select the **Azure Storage** archive type, and the Azure tenant and client for the Datadog App that has the Storage Blob Data Contributor role on your storage account.- Input your storage account name and the container name for your archive.**Optional**: Input a prefix directory for all the content of your log archives. | | **Google Cloud Storage** | - Select the **Google Cloud Storage** archive type, and the GCS Service Account that has permissions to write on your storage bucket.- Input your bucket name.**Optional**: Input a prefix directory for all the content of your log archives. | ### Advanced settings{% #advanced-settings %} {% image source="https://datadog-docs.imgix.net/images/logs/archives/log_archives_advanced_settings.702f81e870c17457c9623f0b645e6b35.png?auto=format" alt="Advanced settings to add optional tags and define max scan size" /%} #### Datadog tags{% #datadog-tags %} Use this optional configuration step to: - Include all log tags in your archives (activated by default on all new archives). **Note**: This increases the size of resulting archives. - Add tags on rehydrated logs according to your Restriction Queries policy. See the [`logs_read_data`](https://docs.datadoghq.com/account_management/rbac/permissions#logs_read_data) permission. #### Define maximum scan size{% #define-maximum-scan-size %} Use this optional configuration step to define the maximum volume of log data (in GB) that can be scanned for Rehydration on your Log Archives. For Archives with a maximum scan size defined, all users need to estimate the scan size before they are allowed to start a Rehydration. If the estimated scan size is greater than what is permitted for that Archive, users must reduce the time range over which they are requesting the Rehydration. Reducing the time range will reduce the scan size and allow the user to start a Rehydration. #### Archive Partition Attribute (Preview){% #archive-search-partition-attribute %} {% callout %} ##### Join the Preview! Archive Search is in Preview. Request access to search archived logs in real time. No rehydrating, no delays. Instantly access years of data when you need it. [Request Access](https://www.datadoghq.com/product-preview/flex-frozen-archive-search/) {% /callout %} To optimize how your archived logs are physically organized in storage (and accelerate [Archive Search](https://docs.datadoghq.com/logs/log_configuration/archive_search/?tab=amazons3)), configure partition attributes in your Datadog Archive. - **Partition Attributes**: Add low-cardinality attributes such as `service`, `source`, `env`, or `status` that you frequently use as search filters. - **Benefit**: Logs sharing the same partition attribute values are co-located in storage. When searching, Datadog can skip entire partitions that don't match your query, drastically reducing the volume of data scanned. #### Archive Lookup Attribute (Preview){% #archive-search-lookup-attribute %} {% callout %} ##### Join the Preview! Archive Search is in Preview. Request access to search archived logs in real time. No rehydrating, no delays. Instantly access years of data when you need it. [Request Access](https://www.datadoghq.com/product-preview/flex-frozen-archive-search/) {% /callout %} To accelerate searches and investigations in your archives (with [Archive Search](https://docs.datadoghq.com/logs/log_configuration/archive_search/?tab=amazons3)), configure lookup attributes in your Datadog Archive. - **Lookup Attributes**: Add high-cardinality attributes such as `trace_id`, `container_id`, or `customer_id`. - **Benefit**: This allows you to pinpoint specific logs within your long-term storage much faster, reducing the time and data scanned during ad-hoc investigations. **Partition vs. Lookup attributes** | Partition | Lookup | | ---------------------- | ----------------------------------------- | | **Cardinality** | Low (tens to hundreds of values) | High (millions of values) | | **Typical attributes** | `service`, `source`, `env`, `status` | `trace_id`, `container_id`, `user_id`, `transaction_id` | | **How it helps** | Prunes entire partitions from scan | Pinpoints individual log entries within your archive | | **Best used for** | Broad filtering by environment or service | Ad-hoc investigations on specific identifiers | For maximum search performance, combine both: partition attributes narrow the search scope to the relevant data segments, while lookup attributes let you find specific logs within those segments instantly. {% callout %} # Important note for users on the following Datadog sites: us3.datadoghq.com #### Firewall rules{% #firewall-rules %} {% tab title="Azure storage" %} Firewall rules are not supported. {% /tab %} {% /callout %} #### Storage class{% #storage-class %} {% tab title="AWS S3" %} You can either select a storage class for your archive or [set a lifecycle configuration on your S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-set-lifecycle-configuration-intro.html) to automatically transition your log archives to optimal storage classes. [Rehydration](https://docs.datadoghq.com/logs/archives/rehydrating/) only supports the following storage classes: - S3 Standard - S3 Standard-IA - S3 One Zone-IA - S3 Glacier Instant Retrieval - S3 Intelligent-Tiering, only if [the optional asynchronous archive access tiers](https://aws.amazon.com/s3/storage-classes/intelligent-tiering/) are both disabled. If you wish to rehydrate from archives in another storage class, you must first move them to one of the supported storage classes above. {% /tab %} {% tab title="Azure Storage" %} Archiving and [Rehydration](https://docs.datadoghq.com/logs/archives/rehydrating/) only supports the following access tiers: - Hot access tier - Cool access tier If you wish to rehydrate from archives in another access tier, you must first move them to one of the supported tiers above. {% /tab %} {% tab title="Google Cloud Storage" %} Archiving and [Rehydration](https://docs.datadoghq.com/logs/archives/rehydrating/) supports the following access tiers: - Standard - Nearline - Coldline - Archive {% /tab %} #### Server-side encryption (SSE) for S3 archives{% #server-side-encryption-sse-for-s3-archives %} When creating or updating an S3 archive in Datadog, you can optionally configure **Advanced Encryption**. Three options are available under the **Encryption Type** dropdown: - **Default S3 Bucket-Level Encryption** (Default): Datadog does not override your S3 bucket's default encryption settings. - **Amazon S3 managed keys**: Forces server-side encryption using Amazon S3 managed keys ([SSE-S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html)), regardless of the S3 bucket's default encryption. - **AWS Key Management Service**: Forces server-side encryption using a customer-managed key (CMK) from [AWS KMS](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html), regardless of the S3 bucket's default encryption. You will need to provide the CMK ARN. {% tab title="Default S3 Bucket-Level Encryption" %} When this option is selected, Datadog does not specify any encryption headers in the upload request. The default encryption from your S3 bucket will apply. To set or check your S3 bucket's encryption configuration: 1. Navigate to your S3 bucket. 1. Click the **Properties** tab. 1. In the **Default Encryption** section, configure or confirm the encryption type. If your encryption uses [AWS KMS](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html), ensure that you have a valid CMK and CMK policy attached to your CMK. {% /tab %} {% tab title="Amazon S3 managed keys" %} This option ensures that all archives objects are uploaded with [SSE_S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html), using Amazon S3 managed keys. This overrides any default encryption setting on the S3 bucket. {% /tab %} {% tab title="AWS Key Management Service" %} This option ensures that all archives objects are uploaded using a customer-managed key (CMK) from [AWS KMS](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). This overrides any default encryption setting on the S3 bucket. Ensure that you have completed the following steps to create a valid CMK and CMK policy. Then, provide the CMK ARN to successfully configure this encryption type. 1. Create your CMK. 1. Attach a CMK policy to your CMK with the following content, replacing the AWS account number and Datadog IAM role name appropriately: ``` { "Id": "key-consolepolicy-3", "Version": "2012-10-17", "Statement": [ { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::root" }, "Action": "kms:*", "Resource": "*" }, { "Sid": "Allow use of the key", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ], "Resource": "*" }, { "Sid": "Allow attachment of persistent resources", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/" }, "Action": [ "kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant" ], "Resource": "*", "Condition": { "Bool": { "kms:GrantIsForAWSResource": "true" } } } ] } ``` After selecting **AWS Key Management Service** as your **Encryption Type** in Datadog, input your AWS KMS key ARN. {% /tab %} ### Validation{% #validation %} Once your archive settings are successfully configured in your Datadog account, your processing pipelines begin to enrich all logs ingested into Datadog. These logs are subsequently forwarded to your archive. However, after creating or updating your archive configurations, it can take several minutes before the next archive upload is attempted. The frequency at which archives are uploaded can vary. **Check back on your storage bucket in 15 minutes** to make sure the archives are successfully being uploaded from your Datadog account. After that, if the archive is still in a pending state, check your inclusion filters to make sure the query is valid and matches log events in [Live Tail](https://docs.datadoghq.com/logs/explorer/live_tail/). When Datadog fails to upload logs to an external archive, due to unintentional changes in settings or permissions, the corresponding Log Archive is highlighted in the configuration page. {% image source="https://datadog-docs.imgix.net/images/logs/archives/archive_errors_details.ee33beebe9724e5cb6acdca3083c5b06.png?auto=format" alt="Check that your archives are properly set up" /%} Hover over the archive to view the error details and the actions to take to resolve the issue. An event is also generated in the [Events Explorer](https://docs.datadoghq.com/events/explorer/). You can create a monitor for these events to detect and remediate failures quickly. ## Multiple archives{% #multiple-archives %} If multiple archives are defined, logs enter the first archive based on filter. {% image source="https://datadog-docs.imgix.net/images/logs/archives/log_forwarding_archives_multiple.688f9d3f3e1b93f95b0b0efb9e30e2e9.png?auto=format" alt="Logs enter the first archive whose filter they match on." /%} It is important to order your archives carefully. For example, if you create a first archive filtered to the `env:prod` tag and a second archive without any filter (the equivalent of `*`), all your production logs would go to one storage bucket or path, and the rest would go to the other. ## Format of the archives{% #format-of-the-archives %} The log archives that Datadog forwards to your storage bucket are in compressed JSON format (`.json.gz`). Using the prefix you indicate (or `/` if there is none), the archives are stored in a directory structure that indicates on what date and at what time the archive files were generated, such as the following: ``` /my/bucket/prefix/dt=20180515/hour=14/archive_143201.1234.02aafad5-f525-4592-905e-e962d1a5b2f7.json.gz /my/bucket/prefix/dt=/hour=/archive_..json.gz ``` This directory structure simplifies the process of querying your historical log archives based on their date. ## Further Reading{% #further-reading %} - [Learn how to access your archived log content in Datadog](https://docs.datadoghq.com/logs/archives/rehydrating) - [Learn about the Log Explorer](https://docs.datadoghq.com/logs/explorer/) - [Learn about Logging without Limits*](https://docs.datadoghq.com/logs/logging_without_limits/) \*Logging without Limits is a trademark of Datadog, Inc.