---
title: Generate Log-based Metrics Processor
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: >-
  Docs > Observability Pipelines > Processors > Generate Log-based Metrics
  Processor
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Generate Log-based Metrics Processor

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}
Available for:
{% icon name="icon-logs" /%}
 Logs 
## Overview{% #overview %}

Many types of logs are meant to be used for telemetry to track trends, such as KPIs, over long periods of time. Generating metrics from your logs is a cost-effective way to summarize log data from high-volume logs, such as CDN logs, VPC flow logs, firewall logs, and network logs. Use the Generate Metrics processor to generate count, gauge, or distribution metrics from logs that match a query, and send the metrics to your destination.

**Note**: The metrics generated from logs and routed to Datadog are [custom metrics](https://docs.datadoghq.com/metrics/custom_metrics.md) and billed accordingly. See [Custom Metrics Billing](https://docs.datadoghq.com/account_management/billing/custom_metrics.md) for more information.

## Setup{% #setup %}

To set up the processor:

Click Manage Metrics to create new metrics or edit existing metrics. This opens a side panel.

- If you have not created any metrics yet, enter the metric parameters as described in the Add a metric section to create a metric.
- If you have already created metrics, click on the metric's row in the overview table to edit or delete it. Use the search bar to find a specific metric by its name, and then select the metric to edit or delete it. Click Add Metric to add another metric.

### Add a metric{% #add-a-metric %}

{% alert level="warning" %}
The Generate Metrics processor uses the `timestamp` field on a log to set the metric's timestamp. If the log `timestamp` field is a string value, the log's processing time is used instead. See Convert string timestamp to timestamp format for more information.
{% /alert %}

1. Enter a filter query. Only logs that match the specified filter query are processed. All logs, regardless of whether they match the filter query, are sent to the next step in the pipeline. See [Search Syntax](https://docs.datadoghq.com/observability_pipelines/search_syntax/logs.md) for more information. **Note**: Since a single processor can generate multiple metrics, you can define a different filter query for each metric.
1. Enter a name for the metric.
1. In the Define parameters section, select the metric type (count, gauge, or distribution). See the Count metric example and Distribution metric example. Also see Metrics Types for more information.
   - For gauge and distribution metric types, select a log field which has a numeric (or parseable numeric string) value that is used for the value of the generated metric.
   - For the distribution metric type, the log field's value can be an array of (parseable) numerics, which is used for the generated metric's sample set.
   - The Group by field determines how the metric values are grouped together. For example, if you have hundreds of hosts spread across four regions, grouping by region allows you to graph one line for every region. The fields listed in the Group by setting are set as tags on the configured metric.
1. Click Add Metric.

### Configure a metrics destination{% #configure-a-metrics-destination %}

{% callout %}
# Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com

{% callout %}
##### Join the Preview!

Sending metrics generated from logs to the Splunk HEC, Elasticsearch, or HTTP/S Client destination is in Preview. Contact your account manager to request access.
{% /callout %}

{% /callout %}

{% alert level="info" %}
The option to send generated metrics to a destination other than [Datadog Metrics](https://docs.datadoghq.com/observability_pipelines/destinations/datadog_metrics.md) is available for Worker versions 2.18 and later.  If you upgrade to Worker version 2.18 or later for an existing pipeline that already has a Generate Metrics processor and you want to select a destination other than Datadog Metrics, you must:     1. Delete the previous Generate Metrics processor.     2. Add and configure a new Generate Metrics processor.
{% /alert %}

{% image
   source="https://docs.dd-static.net/images/observability_pipelines/processors/generate_metrics_destination.9e23fb8503a183cfeec79dfa65c1de31.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/processors/generate_metrics_destination.9e23fb8503a183cfeec79dfa65c1de31.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Generate Metrics processor with the select a destination highlighted." /%}

1. On the Generate Metrics processor, click **Add Metrics Destination**.**Note**: If you are using Pipeline Simulation, return to the pipeline page to configure your metrics destination. Click **Back to pipeline** on the top-right corner of the Pipeline Simulation page.
1. [Datadog Metrics](https://docs.datadoghq.com/observability_pipelines/destinations/datadog_metrics.md) is the default destination. To select a different destination, click the pencil icon on the Datadog Metrics destination and select **Change metrics destination**.
1. Select your destination and follow the setup instructions for the specific [destination](https://docs.datadoghq.com/observability_pipelines/destinations.md?tab=metrics#destinations).

## Metrics types{% #metrics-types %}

You can generate these types of metrics for your logs. See the [Metrics types](https://docs.datadoghq.com/metrics/types.md) and [Distributions](https://docs.datadoghq.com/metrics/distributions.md) documentation for more details.

| Metric type  | Description                                                                                                                                         | Example                                                                                       |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| COUNT        | The total number of event occurrences in one time interval. Can be reset to zero, but cannot be decreased.                                          | You want to count the number of logs with `status:error`.                                     |
| GAUGE        | A snapshot of a value at the time it is reported.                                                                                                   | You want to track the latest CPU utilization per host.                                        |
| DISTRIBUTION | Raw values sent to Datadog so percentile aggregations (such as p95, p99) are computed server-side, globally across every host reporting the metric. | You want the global p95 of `response_time_seconds` across every host serving an API endpoint. |

### Count metric example{% #count-metric-example %}

For this `status:error` log example:

```
{"status": "error", "env": "prod", "host": "ip-172-25-222-111.ec2.internal"}
```

To create a count metric that counts the number of logs that contain `"status":"error"` and groups them by `env` and `host`, enter the following information:

| Input parameters | Value                |
| ---------------- | -------------------- |
| Filter query     | `@status:error`      |
| Metric name      | `status_error_total` |
| Metric type      | Count                |
| Group by         | `env`, `prod`        |

### Distribution metric example{% #distribution-metric-example %}

For this example of an API response log:

```
{
    "timestamp": "2018-10-15T17:01:33Z",
    "method": "GET",
    "status": 200,
    "request_body": "{"information"}",
    "response_time_seconds: 10
}
```

To create a distribution metric that measures the average time it takes for an API call to be made, enter the following information:

| Input parameters       | Value                   |
| ---------------------- | ----------------------- |
| Filter query           | `@method`               |
| Metric name            | `status_200_response`   |
| Metric type            | Distribution            |
| Select a log attribute | `response_time_seconds` |
| Group by               | `method`                |

## Convert string timestamp to timestamp format{% #convert-string-timestamp-to-timestamp-format %}

The Generate Metrics processor can only use the log `timestamp` field to set the metric timestamp if the log field is a timestamp type. If the `timestamp` field is a string, the time when the log is processed is used instead. To use the log `timestamp`, you must convert the string to a timestamp type before sending logs to the Generate Metrics processor.

To convert a string timestamp to timestamp format:

1. Add a [Custom Processor](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor.md#setup) to your pipeline before the Generate Metrics processor.
1. Add a function with the following custom script:
   ```
   .timestamp = parse_timestamp!(.timestamp, format: "%+")
   ```
See [parse_timestamp](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor.md#parse_timestamp) for more information.