Custom Metrics

Custom Metrics

Overview

There are a few different ways to submit custom metrics to Datadog from a Lambda function.

  • Creating custom metrics from logs or traces: If your Lambda functions are already sending trace or log data to Datadog, and the data you want to query is captured in an existing log or trace, you can generate custom metrics from logs and traces without re-deploying or making any changes to your application code.
  • Submitting custom metrics using the Datadog Lambda Extension: If you want to submit custom metrics directly from your Lambda function, Datadog recommends using the Datadog Lambda Extension. Check whether the Datadog Lambda Extension is supported in your Lambda function runtime.
  • Submitting custom metrics using the Datadog Forwarder Lambda: If you want to submit custom metrics from a runtime that is not yet supported by the Datadog Lambda Extension, you can use the Datadog Forwarder Lambda.
  • (Deprecated) Submitting custom metrics from CloudWatch logs: The method to submit custom metrics by printing a log formatted as MONITORING|<UNIX_EPOCH_TIMESTAMP>|<METRIC_VALUE>|<METRIC_TYPE>|<METRIC_NAME>|#<TAG_LIST> has been deprecated, and you should migrate to one of the solutions above.
  • (Not recommended) Using a third-party library: Most third-party libraries do not submit metrics as distributions and can lead to under-counted results.

Understanding distribution metrics

Custom metrics submitted from Lambda functions are aggregated as distributions, because they are designed to instrument applications, independent of the underlying hosts. You can query the metrics using aggregations: avg, sum, max, min, count. You can also enable percentile aggregations (p50, p75, p90, p95, p99), and manage tags for aggregation on the Metric Summary page.

Creating custom metrics from logs or traces

With log-based metrics, you can record a count of logs that match a query or summarize a numeric value contained in a log, such as request duration. Log-based metrics are a cost-efficient way to summarize log data from the entire ingest stream. Learn more about creating log-based metrics here.

You can also generate metrics from 100% of ingested spans, regardless of whether they are indexed by a retention filter. Learn more about creating span-based metrics here.

With the Datadog Lambda Extension

Datadog recommends using the Datadog Lambda Extension to submit custom metrics from supported Lambda runtimes.

  1. Follow the general serverless installation instructions to configure your Lambda function and install the Datadog Lambda Library and Extension.
  2. If you are not interested in collecting traces from your Lambda function, set the environment variable DD_TRACE_ENABLED to false.
  3. If you are not interested in collecting logs from your Lambda function, set the environment variable DD_SERVERLESS_LOGS_ENABLED to false.
  4. Import and use the helper function from the Datadog Lambda Library, such as lambda_metric or sendDistributionMetric, to submit your custom metrics following the sample code.

If your Lambda function is running in a VPC, ensure that your function can reach Datadog API endpoints either through the public internet, PrivateLink or a proxy.

With the Datadog Forwarder

Datadog recommends using the Datadog Forwarder Lambda to submit custom metrics from Lambda runtimes that are not yet supported by the Datadog Lambda Extension.

  1. Follow the general serverless installation instructions to configure your Lambda function, install the Datadog Lambda Library and the Datadog Forwarder Lambda function, and subscribe the Forwarder to your function’s log group.
  2. If you are not interested in collecting traces from your Lambda function, set the environment variable DD_TRACE_ENABLED to false on your own Lambda function.
  3. If you are not interested in collecting logs from your Lambda function, set the Forwarder’s CloudFormation stack parameter DdForwardLog to false.
  4. Import and use the helper function from the Datadog Lambda Library, such as lambda_metric or sendDistributionMetric, to submit your custom metrics following the sample code.

If the Datadog Lambda Library is not available for your runtime, you can print metrics to CloudWatch logs in the expected JSON format on your own. Select the “Other” tab from the sample code section.

Custom metrics sample code

Note: The arguments to the custom metrics reporting methods have the following requirements:

  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.
  • <METRIC_VALUE> MUST be a number (that is, integer or float).
  • <TAG_LIST> is optional and formatted, for example: ['owner:Datadog', 'env:demo', 'cooltag'].
from datadog_lambda.metric import lambda_metric

def lambda_handler(event, context):
    lambda_metric(
        "coffee_house.order_value",             # Metric name
        12.45,                                  # Metric value
        tags=['product:latte', 'order:online']  # Associated tags
    )
const { sendDistributionMetric } = require('datadog-lambda-js');

async function myHandler(event, context) {
    sendDistributionMetric(
        'coffee_house.order_value', // Metric name
        12.45, // Metric value
        'product:latte', // First tag
        'order:online' // Second tag
    );
    return {
        statusCode: 200,
        body: 'hello, dog!'
    };
}
package main

import (
  "github.com/aws/aws-lambda-go/lambda"
  "github.com/DataDog/datadog-lambda-go"
)

func main() {
  // You only need to wrap your function handler (Not helper functions). 
  lambda.Start(ddlambda.WrapHandler(myHandler, nil))
  /* OR with manual configuration options
  lambda.Start(ddlambda.WrapHandler(myHandler, &ddlambda.Config{
    BatchInterval: time.Second * 15
    APIKey: "my-api-key",
  }))
  */
}

func myHandler(ctx context.Context, event MyEvent) (string, error) {
  ddlambda.Distribution(
    "coffee_house.order_value",     // Metric name
    12.45,                          // Metric value
    "product:latte", "order:online" // Associated tags
  )
  // ...
}
require 'datadog/lambda'

def handler(event:, context:)
    # You only need to wrap your function handler (Not helper functions).
    Datadog::Lambda.wrap(event, context) do
        Datadog::Lambda.metric(
          'coffee_house.order_value',         # Metric name
          12.45,                              # Metric value
          "product":"latte", "order":"online" # Associated tags
        )
        return { statusCode: 200, body: 'Hello World' }
    end
end
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
    public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
        DDLambda dd = new DDLambda(request, lambda);

        Map<String,String> myTags = new HashMap<String, String>();
            myTags.put("product", "latte");
            myTags.put("order", "online");
        
        dd.metric(
            "coffee_house.order_value", // Metric name
            12.45,                      // Metric value
            myTags);                    // Associated tags
    }
}

Write a reusable function that logs your custom metrics in the following format:

{
    "m": "Metric name",
    "v": "Metric value",
    "e": "Unix timestamp (seconds)",
    "t": "Array of tags"
}

For example:

{
    "m": "coffee_house.order_value",
    "v": 12.45,
    "e": 1572273854,
    "t": ["product:latte", "order:online"]
}

[DEPRECATED] CloudWatch logs

This method of submitting custom metrics is no longer supported, and is disabled for all new customers. Migrate to one of the recommended solutions.

Note: If you are migrating to one of the recommended solutions, you’ll need to start instrumenting your custom metrics under new metric names when submitting them to Datadog. The same metric name cannot simultaneously exist as both distribution and non-distribution metric types.

This requires the following AWS permissions in your [Datadog IAM policy][0].

AWS PermissionDescription
logs:DescribeLogGroupsList available log groups.
logs:DescribeLogStreamsList available log streams for a group.
logs:FilterLogEventsFetch specific log events for a stream to generate metrics.

To send custom metrics to Datadog from your Lambda logs, print a log line using the following format:

MONITORING|<UNIX_EPOCH_TIMESTAMP>|<METRIC_VALUE>|<METRIC_TYPE>|<METRIC_NAME>|#<TAG_LIST>

Where:

  • MONITORING signals to the Datadog integration that it should collect this log entry.
  • <UNIX_EPOCH_TIMESTAMP> is in seconds, not milliseconds.
  • <METRIC_VALUE> MUST be a number (that is, integer or float).
  • <METRIC_TYPE> is count, gauge, histogram, or check.
  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.
  • <TAG_LIST> is optional, comma separated, and must be preceded by #. The tag function_name:<name_of_the_function> is automatically applied to custom metrics.

Note: The sum for each timestamp is used for counts and the last value for a given timestamp is used for gauges. It is not recommended to print a log statement every time you increment a metric, as this increases the time it takes to parse your logs. Continually update the value of the metric in your code, and print one log statement for that metric before the function finishes.