---
title: Best Practices for Log Management
description: Best practices for managing and monitoring your logs efficiently in Datadog.
breadcrumbs: Docs > Log Management > Logs Guides > Best Practices for Log Management
---

# Best Practices for Log Management

## Overview{% #overview %}

Datadog Log Management collects, processes, archives, explores, and monitors your logs, so that you have visibility into your system's issues. However, it can be hard to get the right level of visibility from your logs and log throughput can vary highly, creating unexpected resource usage.

Therefore, this guide walks you through various Log Management best practices and account configurations that provide you flexibility in governance, usage attribution, and budget control. More specifically, how to:

- Set up multiple indexes to segment your logs
- Set up multiple archives for long-term storage
- Set up RBAC for custom roles

This guide also goes through how to monitor your log usage by:

- Alerting on unexpected log traffic spikes
- Alerting on indexed logs when the volume passes a specified threshold
- Monitoring which indexes are queried actively
- Setting up exclusion filters on high-volume logs

If you want to transform your logs or redact sensitive data in your logs before they leave your environment, see how to [aggregate, process, and transform your log data with Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/).

## Log account configuration{% #log-account-configuration %}

### Set up multiple indexes for log segmentation{% #set-up-multiple-indexes-for-log-segmentation %}

Set up multiple indexes if you want to segment your logs for different retention periods or daily quotas, usage monitoring, and billing.

For example, if you have logs that only need to be retained for 7 days, while other logs need to be retained for 30 days, use multiple indexes to separate out the logs by the two retention periods.

To set up multiple indexes:

1. Navigate to [Log Indexes](https://app.datadoghq.com/logs/pipelines/indexes).
1. Click **New Index** or **Add a new index**.
1. Enter a name for the Index.
1. Enter the search query to filter to the logs you want in this index.
1. Set the daily quota to limit the number of logs that are stored within an index per day.
1. Set the retention period to how long you want to retain these logs.
1. Click **Save**.

Setting daily quotas on your indexes can help prevent billing overages when new log sources are added or if a developer unintentionally changes the logging levels to debug mode. See Alert on indexes reaching their daily quota on how to set up a monitor to alert when a percentage of the daily quota is reached within the past 24 hours.

### Set up storage for long-term retention{% #set-up-storage-for-long-term-retention %}

If you want to retain logs for an extended time while maintaining querying speeds similar to Standard Indexing, configure [Flex Logs](https://docs.datadoghq.com/logs/log_configuration/flex_logs/). This tier is best suited for logs that require longer retention and occasionally need to be queried urgently. Flex Logs decouples storage from compute costs so you can cost effectively retain more logs for longer without sacrificing visibility. Logs that need to be frequently queried should be stored in standard indexes.

### Set up multiple archives for long-term storage{% #set-up-multiple-archives-for-long-term-storage %}

If you want to store your logs for longer periods of time, set up [Log Archives](https://docs.datadoghq.com/logs/log_configuration/archives/) to send your logs to a storage-optimized system, such as Amazon S3, Azure Storage, or Google Cloud Storage. When you want to use Datadog to analyze those logs, use [Log Rehydration](https://docs.datadoghq.com/logs/log_configuration/rehydrating/)™ to capture those logs back in Datadog. With multiple archives, you can both segment logs for compliance reasons and keep rehydration costs under control.

#### Set up max scan size to manage expensive rehydrations{% #set-up-max-scan-size-to-manage-expensive-rehydrations %}

Set a limit on the volume of logs that can be rehydrated at one time. When setting up an archive, you can define the maximum volume of log data that can be scanned for Rehydration. See [Define maximum scan size](https://docs.datadoghq.com/logs/log_configuration/archives/?tab=awss3#define-maximum-scan-size) for more information.

### Set up RBAC for custom roles{% #set-up-rbac-for-custom-roles %}

There are three [default Datadog roles](https://docs.datadoghq.com/account_management/rbac/?tab=datadogapplication#datadog-default-roles): Admin, Standard, and Read-only. You can also create custom roles with unique permission sets. For example, you can create a role that restricts users from modifying index retention policies to avoid unintended cost spikes. Similarly, you can restrict who can modify log parsing configurations to avoid unwanted changes to well-defined log structures and formats.

To set up custom roles with permissions:

1. Log in to [Datadog](https://app.datadoghq.com/) as an Admin.
1. Navigate to [Organization Settings > Roles](https://app.datadoghq.com/organization-settings/roles).
1. To enable custom roles, click the cog on the top left and then click **Enable**.
1. Once enabled, click **New Role**.
1. Enter a name for the new role.
1. Select the permissions for the role. This allows you to restrict access to certain actions, such as rehydrating logs and creating log-based metrics. See [Log Management Permissions](https://docs.datadoghq.com/account_management/rbac/permissions/?tab=ui#log-management) for details.
1. Click **Save**.

See [How to Set Up RBAC for Logs](https://docs.datadoghq.com/logs/guide/logs-rbac/) for a step-by-step guide on how to set up and assign a role with specific permissions for an example use case.

## Monitor log usage{% #monitor-log-usage %}

You can monitor your log usage, by setting up the following:

- Alerts for unexpected log traffic spikes
- Alert when an indexed log volume passes a specified threshold

### Alert on unexpected log traffic spikes{% #alert-on-unexpected-log-traffic-spikes %}

#### Log usage metrics{% #log-usage-metrics %}

By default, [log usage metrics](https://docs.datadoghq.com/logs/logs_to_metrics/#logs-usage-metrics) are available to track the number of ingested logs, ingested bytes, and indexed logs. These metrics are free and kept for 15 months:

- `datadog.estimated_usage.logs.ingested_bytes`
- `datadog.estimated_usage.logs.ingested_events`

See [Anomaly detection monitors](https://docs.datadoghq.com/monitors/types/anomaly/) for steps on how to create anomaly monitors with the usage metrics.

**Note**: Datadog recommends setting the unit to `byte` for the `datadog.estimated_usage.logs.ingested_bytes` in the [metric summary page](https://app.datadoghq.com/metric/summary?filter=datadog.estimated_usage.logs.ingested_bytes&metric=datadog.estimated_usage.logs.ingested_bytes):

{% image
   source="https://datadog-docs.imgix.net/images/logs/guide/logs_estimated_bytes_unit.af095f3911c9417ef7c99b6bc933d715.png?auto=format"
   alt="The metric summary page showing the datadog.estimated_usage.logs.ingested_bytes side panel with the unit set to byte" /%}

#### Anomaly detection monitors{% #anomaly-detection-monitors %}

Create an anomaly detection monitor to alert on any unexpected log indexing spikes:

1. Navigate to [Monitors > New Monitor](https://app.datadoghq.com/monitors/create) and select **Anomaly**.
1. In the **Define the metric** section, select the `datadog.estimated_usage.logs.ingested_events` metric.
1. In the **from** field, add the `datadog_is_excluded:false` tag to monitor indexed logs and not ingested ones.
1. In the **sum by** field, add the `service` and `datadog_index` tags, so that you are notified if a specific service spikes or stops sending logs in any index.
1. Set the alert conditions to match your use case. For example, set the monitor to alert if the evaluated values are outside of an expected range.
1. Add a title for the notification and a message with actionable instructions. For example, this is a notification with contextual links:
   ```text
   An unexpected amount of logs has been indexed in the index: {{datadog_index.name}}
   
   1. [Check Log patterns for this service](https://app.datadoghq.com/logs/patterns?from_ts=1582549794112&live=true&to_ts=1582550694112&query=service%3A{{service.name}})
   2. [Add an exclusion filter on the noisy pattern](https://app.datadoghq.com/logs/pipelines/indexes)
   ```
1. Click **Create**.

### Alert when an indexed log volume passes a specified threshold{% #alert-when-an-indexed-log-volume-passes-a-specified-threshold %}

Set up a monitor to alert if an indexed log volume in any scope of your infrastructure (for example, `service`, `availability-zone`, and so forth) is growing unexpectedly.

1. Navigate to the [Log Explorer](https://app.datadoghq.com/logs).
1. Enter a [search query](https://docs.datadoghq.com/logs/explorer/search/) that includes the index name (for example, `index:main`) to capture the log volume you want to monitor.
1. Click **More…** and select **Create monitor**.
1. Add tags (for example, `host,`services, and so on) to the **group by** field.
1. Enter the **Alert threshold** for your use case. Optionally, enter a **Warning threshold**.
1. Add a notification title, for example:
   ```
   Unexpected spike on indexed logs for service {{service.name}}
   ```
1. Add a message, for example:
   ```
   The volume on this service exceeded the threshold. Define an additional exclusion filter or increase the sampling rate to reduce the volume.
   ```
1. Click **Create**.

#### Alert on indexed logs volume since the beginning of the month{% #alert-on-indexed-logs-volume-since-the-beginning-of-the-month %}

Leverage the `datadog.estimated_usage.logs.ingested_events` metric filtered on `datadog_is_excluded:false` to only count indexed logs and the [metric monitor cumulative window](https://docs.datadoghq.com/monitors/configuration/?tab=thresholdalert#evaluation-window) to monitor the count since the beginning of the month.

{% image
   source="https://datadog-docs.imgix.net/images/logs/guide/monthly_usage_monitor.df9ccd17ae84af8f46cea3e831c0f348.png?auto=format"
   alt="Setup a monitor to alert for the count of indexed logs since the beginning of the month" /%}

#### Alert on indexes reaching their daily quota{% #alert-on-indexes-reaching-their-daily-quota %}

[Set up a daily quota](https://docs.datadoghq.com/logs/indexes/#set-daily-quota) on indexes to prevent indexing more than a given number of logs per day. If an index has a daily quota, Datadog recommends that you set the monitor that notifies on that index's volume to alert when 80% of this quota is reached within the past 24 hours.

An event is generated when the daily quota is reached. These events have the `datadog_index` tag which includes the index name. Therefore, when this event has been generated, you can [create a facet](https://docs.datadoghq.com/events/explorer/facets) on the `datadog_index` tag, so that you can use `datadog_index` in the `group by` step for setting up a multi-alert monitor.

To set up a monitor to alert when the daily quota is reached for an index:

1. Navigate to [Monitors > New Monitor](https://app.datadoghq.com/monitors/create) and click **Event**.
1. Enter: `source:datadog datadog_index:* "daily quota reached"` in the **Define the search query** section. Include `datadog_index:*` to ensure only index related events are selected.
1. In the **Count of** field, add `datadog_index` to group by index. This updates the query to read `Show Count of * by datadog_index (datadog_index)`.
1. For **Evaluate the query over**, select **current day**. For **Starting at**, select the time when indexes reset. This keeps the monitor in alert status until quota reset. This is an example of what the search query looks like when defined in Datadog:
   {% image
      source="https://datadog-docs.imgix.net/images/logs/guide/daily_quota_notification_search_query.7dcc7ef243a5ee09041229a48fb9623a.png?auto=format"
      alt="The Datadog Alert on Index Quota Reached Search Query configuration" /%}
1. In the **Set alert conditions** section, select `above or equal to` and enter `1` for the **Alert threshold**.
1. Add a notification title and message in the **Configure notifications and automations** section. The **Multi Alert** button is automatically selected because the monitor is grouped by `datadog_index(datadog_index)`.
1. Click **Save**.

**Note**: The `datadog_index(datadog_index)` tag is only available when an event has already been generated.

This is an example of what the notification looks like in Slack:

{% image
   source="https://datadog-docs.imgix.net/images/logs/guide/daily_quota_notification.48dca0168d69ca542a55df420b2aec16.png?auto=format"
   alt="A slack notification on the daily quota reached on datadog_index:retention-7" /%}

### Review the estimated usage dashboard{% #review-the-estimated-usage-dashboard %}

Once you begin ingesting logs, an out-of-the-box [dashboard](https://app.datadoghq.com/dash/integration/logs_estimated_usage) summarizing your log usage metrics is automatically installed in your account.

{% image
   source="https://datadog-docs.imgix.net/images/logs/guide/logslight.fb91e2fad779b4e0c72becfcba735bc9.png?auto=format"
   alt="The log estimated usage dashboard showing the breakdown of indexed and ingested in different widgets" /%}

**Note**: The metrics used in this dashboard are estimates and may differ from official billing numbers.

To find this dashboard, go to **Dashboards > Dashboards List** and search for [Log Management - Estimated Usage](https://app.datadoghq.com/dashboard/lists?q=Log+Management+-+Estimated+Usage).

### Monitor which indexes are queried actively{% #monitor-which-indexes-are-queried-actively %}

Monitoring **query activity** helps you evaluate the value of your indexed data and optimize costs. For example, you can identify indexes that are rarely queried to reduce retention or move data to Flex Logs or archives.

To analyze which indexes are actively queried:

1. Navigate to the [Audit Trail](https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&group_by=%40asset.new_value.query.indexes]%28https://app.datadoghq.com/audit-trail?query=%40evt.name%3A%22Log%20Management%22%20%40action%3Aqueried&agg_m=count&agg_m_source=base&agg_q=%40asset.new_value.query.indexes&agg_q_source=base&agg_t=count&audit__diff=unified&cols=log_usr.id%2Clog_action%2Clog_evt.name&fromUser=true&messageDisplay=expanded-md&refresh_mode=sliding&stream_sort=desc&top_n=10&top_o=top&viz=query_table&x_missing=true&from_ts=1768733389060&to_ts=1771325389060&live=true). This link pre-fills the required query and grouping.
1. Verify that the query is set to `@evt.name:"Log Management" @action:queried`.
1. Select the **Table** visualization to view a ranked list of the most and least used indexes for the selected time frame.
1. In the **By** section, group logs by `@asset.new_value.query.indexes`.

### Set up exclusion filters on high-volume logs{% #set-up-exclusion-filters-on-high-volume-logs %}

When your usage monitors alert, you can set up exclusion filters and increase the sampling rate to reduce the volume. See [Exclusion Filters](https://docs.datadoghq.com/logs/log_configuration/indexes/#exclusion-filters) on how to set them up. You can also use [Log Patterns](https://docs.datadoghq.com/logs/explorer/analytics/patterns) to group and identify high-volume logs. Then, in the log pattern's side panel, click **Add Exclusion Filter** to add a filter to stop indexing those logs.

{% image
   source="https://datadog-docs.imgix.net/images/logs/guide/patterns_exclusion.f74081b0c87eb4cebb16353ed933327d.png?auto=format"
   alt="The log explorer page showing the side-panel details of a pattern with the add exclusion filter button at the top" /%}

Even if you use exclusion filters, you can still visualize trends and anomalies over all of your log data using log-based metrics. See [Generate Metrics from Ingested Logs](https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/) for more information.

### Enable Sensitive Data Scanner for Personally Identifiable Information (PII) detection{% #enable-sensitive-data-scanner-for-personally-identifiable-information-pii-detection %}

If you want to prevent data leaks and limit non-compliance risks, use Sensitive Data Scanner to identify, tag, and optionally redact or hash sensitive data. For example, you can scan for credit card numbers, bank routing numbers, and API keys in your logs, APM spans, and RUM events, See [Sensitive Data Scanner](https://docs.datadoghq.com/security/sensitive_data_scanner/) on how to set up scanning rules to determine what data to scan.

**Note**: [Sensitive Data Scanner](https://www.datadoghq.com/pricing/?product=sensitive-data-scanner#sensitive-data-scanner) is a separate billable product.

### Enable Audit Trail to see user activities{% #enable-audit-trail-to-see-user-activities %}

If you want to see user activities, such as who changed the retention of an index or who modified an exclusion filter, enable Audit Trail to see these events. See [Audit Trail Events](https://docs.datadoghq.com/account_management/audit_trail/events/) for a list of platform and product-specific events that are available. To enable and configure Audit Trail, follow the steps in the [Audit Trail documentation](https://docs.datadoghq.com/account_management/audit_trail/).

**Note**: [Audit Trail](https://www.datadoghq.com/pricing/?product=audit-trail#audit-trail) is a separate billable product.

## Further Reading{% #further-reading %}

- [Learn how to process your logs](https://docs.datadoghq.com/logs/log_configuration/processors)
- [Learn more about parsing](https://docs.datadoghq.com/logs/log_configuration/parsing)
- [How to implement log management policies with your teams](https://www.datadoghq.com/blog/log-management-policies/)
- [Best practices for managing Datadog organizations at scale](https://www.datadoghq.com/blog/volkswagen-organizations/)
- [Datadog Tips & Tricks: Improve log utilization with Datadog log exclusion filters](https://www.youtube.com/watch?v=2OEzAE7c2c0&list=PLdh-RwQzDsaM9Sq_fi-yXuzhmE7nOlqLE&index=1&pp=iAQB)