---
isPrivate: true
title: Logs Cost Attribution
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: Docs > Log Management > Logs FAQ > Logs Cost Attribution
---

# Logs Cost Attribution

## Overview{% #overview %}

Datadog provides log usage information through the [Log Estimated Usage dashboard](https://app.datadoghq.com/dash/integration/30602/log-management---estimated-usage), the [Plan and Usage](https://app.datadoghq.com/billing/usage) section in the app, and the available [logs usage metrics](https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/#logs-usage-metrics). However, there might be situations where you want visibility into more granular cost attribution data, such as for specific teams.

This guide walks you through the steps on how to create custom metrics and a dashboard to see your log cost attribution for different teams. You can also use this process for other attributes, such as departments, projects, products, regions, and so on.

1. Configure custom tags.
1. Generate custom log metrics with those tags.
1. Create widgets in a dashboard for the custom log metrics.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/cost_attribution_dashboard.86e08dadc903e091097b8ce89de84c25.png?auto=format"
   alt="A dashboard showing table widgets for usage and cost broken down by teams for ingestion, Sensitive Data Scanner, seven days indexing, and 15 days indexing. " /%}

## Create a new log pipeline{% #create-a-new-log-pipeline %}

Create a new log pipeline that filters to logs for which you want to attribute costs. For this example, filter to the subset of logs that you want to breakdown by team.

1. Navigate to [Logs Pipelines](https://app.datadoghq.com/logs/pipelines).
1. Click **Add a new pipeline**.
1. Enter the filter for logs you want to attribute costs.
1. Enter a name for the pipeline. For example, `Cost attribution by team`.
1. Optionally, add tags and a description.
1. Click **Create**.

Leave the new pipeline at the end of the list of pipelines. This lets the logs go through the other pipelines so that those tags and attributes are created first.

Add all processors you create for this cost attribution example to the new pipeline.

### Add a `team` tag{% #add-a-team-tag %}

Datadog recommends that you use one of these [tagging methods](https://docs.datadoghq.com/getting_started/tagging/#tagging-methods) to add the `team` tag to logs **before ingestion**.

However, if you need to configure the tag during ingestion, follow these steps to create and add a `team` tag.

1. Create a new `team` attribute.
1. Create a remapper to convert the `team` attribute to a tag.

You can use this process to create the attributes you want for breaking down your log usage (for example, by departments, products, regions, and so forth).

#### Create a new `team` attribute{% #create-a-new-team-attribute %}

Use a [Category Processor](https://docs.datadoghq.com/logs/log_configuration/processors/?tab=ui#category-processor) to create a new `team` attribute for your logs.

1. Navigate to the new pipeline and click **Add processor**.
1. Select **Category Processor** for the processor type.
1. Enter a name for the processor. For example, "Create team attribute".
1. Enter `team` in the **Set target category attribute** field. This creates a `team` attribute.
1. In the **Populate category** section, add a category for each team. For example, to add the tag `team:service_a` to log events that match `service:a` and `env:prod`:
   1. Enter `service:a` and `env:prod` in the **All events that match** field.
   1. Enter `service_a` in the **Appear under the value name** field.
   1. Click **Add**.
1. Add the other teams as separate categories.
1. Click **Create**.

#### Create a remapper to convert the `team` attribute to a tag{% #create-a-remapper-to-convert-the-team-attribute-to-a-tag %}

1. Navigate to the pipeline and click **Add processor**.
1. Select **Remapper** for the processor type.
1. Enter a name for the processor. For example, "Convert team attribute to tag".
1. In the **Set attribute(s) or tag key to remap** section, select **Attribute(s)** and enter `team`.
1. In the **Set target attribute or tag key** section, select **Tag key** and enter `team`.
1. Disable **Preserve source attribute** to make sure the attribute is removed and only the tag is kept.
1. Enable **Override on conflict**.
1. Click **Create**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/team_remapper.344d496273592c0fdc5b554b7358a939.png?auto=format"
   alt="The create remapper form showing all the data filled in to create a team remapper" /%}

## Configure custom tags{% #configure-custom-tags %}

Create custom tags so you can organize custom log usage metrics into categories that are relevant to your use case. For this example, create the following tags:

- `retention_period` for indicating the number of days for which logs are retained in Datadog indexes.
- `online_archives` for indicating whether logs have been routed to Online Archives.
- `sds` for indicating whether logs have been scanned by the Sensitive Data Scanner.

### Create a `retention_period` tag{% #create-a-retention_period-tag %}

{% alert level="danger" %}
Datadog recommends that you set up the `retention_period` tag even if your indexes all have the same retention period. This makes sure that if you start using multiple retention periods, all logs are tagged with its retention period.
{% /alert %}

`retention_period` is the number of days your logs are retained in Datadog indexes. Since indexing billing costs are incurred based on the number of days that the logs are retained, use the `retention_period` tag to associate each log with its retention period to see cost attribution.

Datadog recommends using the following method to configure the `retention_period` tag:

1. Create a new `index_name` attribute.
1. Create a new `retention_period` attribute.
1. Create a Remapper to convert the `retention_period` attribute to a tag.

#### Create a new `index_name` attribute{% #create-a-new-index_name-attribute %}

Use a [Category Processor](https://docs.datadoghq.com/logs/log_configuration/processors/?tab=ui#category-processor) to create a new `index_name` attribute for identifying the index to which the logs are routed.

1. Navigate to the pipeline previously created and click **Add processor**.
1. Select **Category Processor** for the processor type.
1. Enter a name for the processor. For example, "Create index_name attribute".
1. Enter **index\_name** in the **Set target category attribute** field. This creates an `index_name` attribute.
1. Add a category for each index. For example, if you have an index named `retention-7` for all logs tagged with `env:staging`:
   {% image
      source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/indexes_configuration.a237e36e04ad10f092deda4a605b265f.png?auto=format"
      alt="The indexes list showing the filter query, retention period, and whether online archives is enabled for the retention-30, retention-15, retention-7, and demo indexes" /%}
Then, in the **Populate category** section:
   1. Enter `env:staging` in the **All events that match** field.
   1. Enter `retention-7` in the **Appear under the value name** field.
   1. Click **Add**.
1. Add the other indexes as separate categories.
1. Click **Create**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/indexes_category_processor.8a5c8a158292d86a49a7e041900d1c21.png?auto=format"
   alt="The category processor form fill in with data to create an index_name attribute" /%}

#### Create a new `retention_period` attribute{% #create-a-new-retention_period-attribute %}

Use a [Category Processor](https://docs.datadoghq.com/logs/log_configuration/processors/?tab=ui#category-processor) to create a new `retention_period` attribute to associate the index with its retention period.

1. Navigate to the pipeline and click **Add processor**.
1. Select **Category Processor** for the processor type.
1. Enter a name for the processor. For example, "Create retention_period attribute".
1. Enter `retention_period` in the **Set target category attribute** field. This creates a `retention_period` attribute.
1. Add a category for each retention period. For example, if you have a 7-day retention index named `retention-7`, then in the **Populate category** section:
   1. Enter `@index_name:(retention-7)` in the **All events that match** field.
   1. Enter `7` in the **Appear under the value name** field.
   1. Click **Add**.
1. Add the other retention periods as separate categories.
1. Click **Create**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/retention_period_processor.e3cc5062b97aa072f64d138799d48e77.png?auto=format"
   alt="The category processor form fill in with data to create a retention_period attribute" /%}

#### Create a remapper to convert the `retention_period` attribute to a tag{% #create-a-remapper-to-convert-the-retention_period-attribute-to-a-tag %}

1. Navigate to the pipeline and click **Add processor**.
1. Select **Remapper** for the processor type.
1. Enter a name for the processor. For example, "Convert retention_period attribute to tag".
1. In the **Set attribute(s) or tag key to remap** section, select **Attribute(s)** and enter `retention_period`.
1. In the **Set target attribute or tag key** section, select **Tag key** and enter `retention_period`.
1. Disable **Preserve source attribute** to make sure the attribute is removed and only the tag is kept.
1. Enable **Override on conflict**.
1. Click **Create**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/retention_period_remapper.2367a873b47295c842fab359e06872af.png?auto=format"
   alt="The create remapper form showing all the data filled in to create a retention_period remapper" /%}

### Create an `online_archives` tag{% #create-an-online_archives-tag %}

{% alert level="danger" %}
Datadog recommends that you set up the `online_archives` tag even if none of your indexes have online archives enabled. This ensures that if you start using Online Archives, the relevant logs are tagged with `online_archives`.
{% /alert %}

The `online_archives` tag indicates whether or not your logs have been routed to Online Archives. Since Online Archives are charged differently than standard indexing, use the `online_archives` tag to determine which logs have been routed to Online Archives and see cost attribution.

Datadog recommends using the following method to configure the `online_archive` tag:

#### Create an `online_archives` attribute{% #create-an-online_archives-attribute %}

Use a [Category Processor](https://docs.datadoghq.com/logs/log_configuration/processors/?tab=ui#category-processor) to create a new `online_archives` attribute to indicate whether or not the associated index has Online Archives enabled.

1. Navigate to the pipeline previously created and click **Add processor**.
1. Select **Category Processor** for the processor type.
1. Enter a name for the processor. For example, "Create online_archives attribute". This creates an `online_archives` attribute.
1. In the **Populate category** section, add two categories:
   - In the **first category**, the value `true` is assigned to all indexes with Online Archives enabled. For example, if logs in the index named `retention-30` go into Online Archives:
     1. Enter `@index_name:(retention-30)` in the **All events that match** field.
     1. Enter `true` in the **Appear under the value name** field.
     1. Click **Add**.
   - In the **second category**, the value `false` is assigned to all other indexes.
     1. Enter `*` in the **All events that match** field.
     1. Enter `false` in the **Appear under the value name** field.
     1. Click **Add**.
1. Click **Create**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/online_archives_attribute.e32dc919a5845cdb0c63acf107e95aac.png?auto=format"
   alt="The category processor form fill in with data to create a online_archives attribute" /%}

#### Create a Remapper to convert the `online_archives` attribute to a tag{% #create-a-remapper-to-convert-the-online_archives-attribute-to-a-tag %}

1. Navigate to the pipeline and click **Add processor**.
1. Select **Remapper** for the processor type.
1. Enter a name for the processor. For example, "Convert online_archives attribute to tag".
1. In the **Set attribute(s) or tag key to remap** section, select **Attribute(s)** and enter `online_archives`.
1. In the **Set target attribute or tag key** section, select **Tag key** and enter `online_archives`.
1. Disable **Preserve source attribute** to make sure the attribute is removed and only the tag is kept.
1. Enable **Override on conflict**.
1. Click **Create**.

{% alert level="info" %}
The order of categories in a Category Processor is important. The attribute is assigned the value of the first category for which the log matches the matching query, with the same logic as the indexes. For this reason, be sure that the matching queries and the order of the index Category Processor are the same as the actual order of the indexes, and that the category `true` is always checked before `false` in the Online Archives Category Processor.If the index configurations are changed, you need to update the processor configuration to reflect the change.
{% /alert %}

Datadog highly recommends automating this process by using the [Datadog API endpoints](https://docs.datadoghq.com/logs/log_configuration/processors/?tab=api#category-processor) to automatically retrieve and update the configuration.

### Create a `sds` tag{% #create-a-sds-tag %}

{% alert level="danger" %}
Datadog recommends that you still set up the `sds` tag even if you are not using the Sensitive Data Scanner. This makes sure that if you start using Sensitive Data Scanner, all the relevant logs are tagged with `sds`.
{% /alert %}

The `sds` tag indicates whether or not your logs have been scanned by the Sensitive Data Scanner. Use the `sds` tag to estimate the costs associated with the specific usage of Sensitive Data Scanner.

For the Sensitive Data Scanner, billed usage is based on the volume of logs scanned, so it matches a scanning group, not a scanning rule. Therefore, you need to create a proxy scanning rule in each scanning group with a regex to match all logs. This ensures that all scanned logs are tagged.

1. Go to the [Sensitive Data Scanner](https://app.datadoghq.com/organization-settings/sensitive-data-scanner).
1. In each scanning group:
   1. Click **Add Scanning Rule**.
   1. Enter `.` in the **Define Regex to match** field to match all logs.
   1. Select **Entire Event** in the **Scan the entire event or a portion of it** field.
   1. Enter `sds:true` in the **Add tags** field.
   1. Leave **Define action on match** on **No action**.
   1. Enter a name for the scanning rule. For example, "Create sds tag".
   1. Click **Create**.

## Generate custom logs metrics{% #generate-custom-logs-metrics %}

Datadog provides a set of [logs usage metrics](https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/#logs-usage-metrics) so that you can see estimated usage. However, since these metrics cannot be modified, you can generate custom logs metrics for your specific log usage cases instead.

Since usage is measured either in gigabytes (GB) or in millions of events depending on the product, you need to generate two different metrics:

- A metric that counts the number of bytes ingested.
- A metric that counts the number of events ingested.

When setting up custom metrics, the tags in the `group by` field are the dimensions of your metric. Use these fields to filter and aggregate the metrics once they are generated. Make sure to include the following tags in the `group by` field:

- `datadog_index`, if the log is routed, the tag contains the name of the index to which the log is routed.
- `datadog_is_excluded` indicates whether the log is rejected by an exclusion filter in the routed index.
- All the custom tags you have configured above (`team`, `retention_period`, `online_archives`, and `sds`).

See [Generate a log-based metric](https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/#generate-a-log-based-metric) for instructions on generating the metrics.

{% alert level="info" %}
It is crucial that you ensure all relevant tags are included in the metric's dimensions because updating a metric's configuration (such as changing the query filters, dimensions, and so on) is not retroactively applied to logs that have already been ingested.
{% /alert %}

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/bytes_injected_metric.7de1aa54c753fb7c60da27b8193a63c9.png?auto=format"
   alt="The new metric form showing logs.estimated.usage.ingested_bytes as the metric name and the group by field with the tags mentioned" /%}

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/events_injected_metric.19be1220b532f1177f42d8611ae9a733.png?auto=format"
   alt="The new metric form showing logs.estimated.usage.ingested_events as the metric name and the group by field with the tags mentioned" /%}

## Create a dashboard using the custom logs metrics{% #create-a-dashboard-using-the-custom-logs-metrics %}

There are several ways to use the generated custom log metrics in Datadog. The metrics can be displayed in dashboards, alerted on, used in Notebooks, queried in the Metrics Explorer, and more.

Datadog recommends that you [create a dashboard](https://docs.datadoghq.com/dashboards/#new-dashboard) with a [table widget](https://docs.datadoghq.com/dashboards/widgets/table/) for each of the following products to track their usage:

- Log Ingestion
- Sensitive Data Scanner for logs
- Log Indexing by retention periods (3, 7, 15, 30 days, and so on)
- Online Archives

To create a new dashboard:

1. Navigate to the [Dashboards list](https://app.datadoghq.com/dashboard/lists).
1. Click **New Dashboard** in the upper right.
1. Enter a dashboard name.
1. Click **New Dashboard**.

### Create a widget for Log Ingestion usage{% #create-a-widget-for-log-ingestion-usage %}

Datadog recommends that you configure the table widget for Log Ingestion in the following way:

1. In the dashboard, click **Add Widgets**.
1. Select the **Table** widget.
1. In the **Metrics** field, select the **bytes** count metric that you generated earlier to count the number of bytes ingested.
1. Select the **sum by** field and add the `team` tag to show the usage in bytes by team. You can also add other tags for your different cost buckets, for example, the `host` tag to see usage by host.
1. Add the following formula to convert usage into costs: `Usage in gigabytes` * `Unit cost for Log Ingestion`.**Note**: If your contractual price per gigabyte changes, you need to update the formula manually.
1. Click **Save**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/logs_ingestion_metric_widget.30fc57a02f9f65bf2540ab7f58c0fb04.png?auto=format"
   alt="The widget edit form showing the data filled in for log ingestion usage" /%}

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/ingestion_widget.7493607894c328739fd5c6766748f41a.png?auto=format"
   alt="The table widget showing the ingestion usage and costs broken down by team" /%}

### Create a widget for Sensitive Data Scanner{% #create-a-widget-for-sensitive-data-scanner %}

Datadog recommends that you configure the table widget for the Sensitive Data Scanner in the following way:

1. In the dashboard, click **Add Widgets**.
1. Select the **Table** widget.
1. In the **Metrics** field, select the **bytes** count metric that you generated earlier to count the number of bytes ingested.
1. In the **from** field, enter `sds:true` to filter only for logs that have been scanned by the Sensitive Data Scanner.
1. Select the **sum by** field and add the `team` tag to show the usage in bytes by team. You can also add other tags for your different cost buckets.
1. Add the following formula to convert usage into costs: `Usage in gigabytes` * `Unit cost for the Sensitive Data Scanner`.**Note**: If your contractual price per gigabyte changes, you need to update the formula manually.
1. Click **Save**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/sds_metric_widget.91abee0959b86dc7a70c4eee3772a6fe.png?auto=format"
   alt="The widget edit form showing the data filled in for Sensitive Data Scanner logs usage" /%}

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/sds_widget.b2577f19fa37950cd84442767fcaf23d.png?auto=format"
   alt="The table widget showing Sensitive Data Scanner usage broken down by team" /%}

### Create a widget for Log Indexing usage{% #create-a-widget-for-log-indexing-usage %}

Since indexing is charged based on the number of days the logs are retained, create one widget for each retention period.

Datadog recommends that you configure the table widget for Log Indexing in the following way:

1. In the dashboard, click **Add Widgets**.
1. Select the **Table** widget.
1. Select the **events** count metric that you generated earlier to count the number of events ingested.
1. In the **from** field, add the following:
   1. `datadog_index:*` to filter to only logs that have been routed to indexes.
   1. `datadog_is_excluded:false` to filter to only logs that have not matched any exclusion filter.
   1. `retention_period:7` to filter to only logs that are retained for 7 days. You don't need to add this tag if you have the same retention period for all your indexes and therefore did not set up this tag earlier. If you have additional `retention_period` tags, create a separate widget for each one.
1. Select the **sum by** field, and add the `team` tag to show the usage in events, by team. You can also add other tags for your different cost buckets.
1. Add the following formula to convert usage into costs: `Usage in millions of events` * `Unit cost for 7 days of retention`. If your contractual price per million of events changes, you need to update the formula manually.
1. Click **Save**.

Create widgets for each `retention_period` tag.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/indexing_metric_widget.d4060fd768bb091c47bd79f0e4a8fbc9.png?auto=format"
   alt="The widget edit form showing the data filled in for log indexing usage" /%}

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/7_day_indexing_widget.6b50f6b8f4c7f1ea3cb9467aaa4de430.png?auto=format"
   alt="The table widget showing 7-day indexing usage broken down by team" /%}

### Create a widget for Online Archives usage{% #create-a-widget-for-online-archives-usage %}

When Online Archives is enabled for an index, logs are duplicated and go into both:

1. Exclusion filters, logs are indexed only if they pass through exclusion filters.
1. Online Archives directly.

Therefore, exclusion filters do not apply to logs that go into the Online Archives.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/exclusion_filters_online_archives.a7c6593db2e906f593306ed2df2499c9.png?auto=format"
   alt="The Online Archive index showing a pipeline for exclusion filters and a pipeline for Online Archives" /%}

Based on that information, Datadog recommends that you configure the table widget for Online Archives in the following way:

1. In the dashboard, click **Add Widgets**.
1. Select the **Table** widget.
1. In the **Metrics** field, select the **events** count metric you generated earlier that counts the number of events ingested.
1. In the **from** field, add the following:
   - `datadog_index:*` to filter to only logs that have been routed to indexes.
   - `online_archives:true` to filter to only logs that have also been routed to Online Archives.
1. Select the **sum by** field and add the `team` tag to show the usage in events by team. You can also add tags for different cost buckets.
1. Add the following formula to convert usage into cost: `Usage in millions of events` * `Unit cost for Online Archives`.**Note**: If your contractual price per million of events changes, you need to update the formula manually.
1. Click **Save**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/online_archives_metric_widget.30edaa30b5d174d5813785b49ba5bf88.png?auto=format"
   alt="The widget edit form showing the data filled in for online archives usage" /%}

### Create a widget for total usage and costs{% #create-a-widget-for-total-usage-and-costs %}

You can aggregate all products into a single widget to get visibility into the total usage and costs. Datadog recommends that you configure the table widget in the following way:

1. In the dashboard, click **Add Widgets**.
1. Select the **Table** widget.
1. Add all queries and formulas created in the other widgets to this widget:
   - Log Ingestion
   - Sensitive Data Scanner for logs
   - Log Indexing
   - Online Archives
1. Click **Save**.

{% image
   source="https://datadog-docs.imgix.net/images/logs/faq/logs_cost_attribution/all_metrics_widget.b79cda6d82ccfdab29a915f8bacbebfa.png?auto=format"
   alt="The graph your data section of the table widget showing six different metrics" /%}

### Further reading{% #further-reading %}

- [Generate metrics from ingested logs](https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/)
- [Learn more about tagging](https://docs.datadoghq.com/getting_started/tagging/)
- [Learn more about the table widget](https://docs.datadoghq.com/dashboards/widgets/table/)