Custom Metrics
New announcements from Dash: Incident Management, Continuous Profiler, and more! New announcements from Dash!

Custom Metrics

Install the Datadog Lambda Library to collect and send custom metrics. Metrics sent from the Datadog Lambda Library are automatically aggregated into distributions, so you can graph the avg, sum, max, min, and count. You can also calculate aggregations over a set of tags for the 50th, 75th, 95th, and 99th percentile values on the Distribution Metrics page.

Distribution metrics are designed to instrument logical objects, like services, independent of the underlying hosts. So, they are well-suited for serverless infrastructure because they aggregate metrics server-side instead of locally with an Agent.

Synchronous vs. Asynchronous Custom Metrics

The Datadog Lambda Library supports submitting custom metrics in Lambda, both synchronously and asynchronously.

Synchronous: The default behavior. This method submits your custom metrics to Datadog via HTTP periodically (every 10 seconds) and at the end of your Lambda invocation. If the invocation lasts for less than 10 seconds, your custom metrics are submitted at the end of the invocation.

Asynchronous (recommended): It’s possible to submit your custom metrics with zero latency overhead and have them appear in Datadog in near-real-time. To accomplish this, the Lambda Library emits your custom metrics as a specially-formatted log, which the Datadog Forwarder parses and submits to Datadog. Logging in AWS Lambda is 100% asynchronous, so this method ensures there is zero latency overhead to your function.

Enabling Asynchronous Custom Metrics

  1. Set the environment variable DD_FLUSH_TO_LOG to True on your Lambda function.
  2. Update your Datadog Forwarder to at least version 1.4.0.

If you are not using Datadog Logs, you can still use asynchronous custom metric submission. Set the environment variable DD_FORWARD_LOG to False on the Datadog log collection AWS Lambda function. This intelligently forwards only custom metrics to Datadog, and not regular logs.

Custom Metrics Sample Code

In your function code, you must import the necessary methods from the Lambda Library and add a wrapper around your function handler. You do not need to wrap your helper functions.

Note: The arguments to the custom metrics reporting methods have the following requirements:

  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.
  • <METRIC_VALUE> MUST be a number (i.e. integer or float).
  • <TAG_LIST> is optional and formatted, for example: ['owner:Datadog', 'env:demo', 'cooltag'].
from datadog_lambda.metric import lambda_metric
from datadog_lambda.wrapper import datadog_lambda_wrapper

# You only need to wrap your function handler (Not helper functions). 
@datadog_lambda_wrapper
def lambda_handler(event, context):
    lambda_metric(
        "coffee_house.order_value",             # Metric name
        12.45,                                  # Metric value
        tags=['product:latte', 'order:online']  # Associated tags
    )
const { datadog, sendDistributionMetric } = require('datadog-lambda-js');

async function myHandler(event, context) {
    sendDistributionMetric(
        'coffee_house.order_value', // Metric name
        12.45, // Metric value
        'product:latte',
        'order:online' // Associated tags
    );
    return {
        statusCode: 200,
        body: 'hello, dog!'
    };
}
// You only need to wrap your function handler (Not helper functions).
module.exports.myHandler = datadog(myHandler);

/* OR with manual configuration options
module.exports.myHandler = datadog(myHandler, {
    apiKey: "my-api-key"
});
*/
package main

import (
  "github.com/aws/aws-lambda-go/lambda"
  "github.com/DataDog/datadog-lambda-go"
)

func main() {
  // You only need to wrap your function handler (Not helper functions). 
  lambda.Start(ddlambda.WrapHandler(myHandler, nil))
  /* OR with manual configuration options
  lambda.Start(ddlambda.WrapHandler(myHandler, &ddlambda.Config{
    BatchInterval: time.Second * 15
    APIKey: "my-api-key",
  }))
  */
}

func myHandler(ctx context.Context, event MyEvent) (string, error) {
  ddlambda.Distribution(
    "coffee_house.order_value",     // Metric name
    12.45,                          // Metric value
    "product:latte", "order:online" // Associated tags
  )
  // ...
}
require 'datadog/lambda'

def handler(event:, context:)
    # You only need to wrap your function handler (Not helper functions).
    Datadog::Lambda.wrap(event, context) do
        Datadog::Lambda.metric(
          'coffee_house.order_value',         # Metric name
          12.45,                              # Metric value
          "product":"latte", "order":"online" # Associated tags
        )
        return { statusCode: 200, body: 'Hello World' }
    end
end
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
    public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
        DDLambda dd = new DDLambda(request, lambda);

        Map<String,String> myTags = new HashMap<String, String>();
            myTags.put("product", "latte");
            myTags.put("order","online");
        
        dd.metric(
            "coffee_house.order_value", // Metric name
            12.45,                      // Metric value
            myTags);                    // Associated tags
    }
}

Emitting asynchronous custom metrics is possible for any language or custom runtime. It works by printing a special JSON-formatted string in your Lambda function that the Datadog Forwarder identifies and submits to Datadog. To use this:

  1. Enable asynchronous cusstom metrics
  2. Write a reusable function that logs your custom metrics in the following format:

    {
    "m": "Metric name",
    "v": "Metric value",
    "e": "Unix timestamp (seconds)",
    "t": "Array of tags"
    }

For example:

{
    "m": "coffee_house.order_value",
    "v": 12.45,
    "e": 1572273854,
    "t": ["product:latte", "order:online"]
}

Note: These custom metrics are submitted as distributions. If you were previously submitting custom metrics another way, consult the documentation on the implications of upgrading to distributions.

Tagging Custom Metrics

You should tag your custom metrics when submitting them with the Datadog Lambda Library. Use the Distribution Metrics page to customize the set of tags applied to your custom metrics.

To add Lambda resource tags to your custom metrics, set the parameter DdFetchLambdaTags to true on the Datadog forwarder CloudFormation stack.

Understanding Distribution Metrics

With distribution metrics, you select the aggregation when graphing or querying it instead of specifying it at submission time.

If you previously submitted custom metrics from Lambda without using one of the Datadog Lambda Librarys, you’ll need to start instrumenting your custom metrics under new metric names when submitting them to Datadog. The same metric name cannot simultaneously exist as both distribution and non-distribution metric types.

To enable percentile aggregations for your distribution metrics, consult the Distribution Metrics page.

Other Submission Methods

Running in a VPC

The Datadog Lambda Library requires access to the public internet to submit custom metrics synchronously. If your Lambda function is associated with a VPC, ensure that it is instead submitting custom metrics asynchronously or that your function can reach the public internet.

Using Third-Party Libraries

There are a number of open source libraries that make it easy to submit custom metrics to Datadog. However, many have not been updated to use Distribution metrics, which are optimized for Lambda. Distribution metrics allow for server-side aggregations independent of a host or locally-running agent. In a serverless environment where there is no agent, Distribution metrics give you flexible aggregations and tagging.

When evaluating third-party metrics libraries for AWS Lambda, ensure they support Distribution metrics.

[DEPRECATED] Using CloudWatch Logs

This method of submitting custom metrics is no longer supported, and is disabled for all new customers. The recommended way to submit custom metrics from Lambda is with a Datadog Lambda Library.

This requires the following AWS permissions in your Datadog IAM policy.

AWS PermissionDescription
logs:DescribeLogGroupsList available log groups.
logs:DescribeLogStreamsList available log streams for a group.
logs:FilterLogEventsFetch specific log events for a stream to generate metrics.

[DEPRECATED] To send custom metrics to Datadog from your Lambda logs, print a log line using the following format:

MONITORING|<UNIX_EPOCH_TIMESTAMP>|<METRIC_VALUE>|<METRIC_TYPE>|<METRIC_NAME>|#<TAG_LIST>

Where:

  • MONITORING signals to the Datadog integration that it should collect this log entry.
  • <UNIX_EPOCH_TIMESTAMP> is in seconds, not milliseconds.
  • <METRIC_VALUE> MUST be a number (i.e. integer or float).
  • <METRIC_TYPE> is count, gauge, histogram, or check.
  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.
  • <TAG_LIST> is optional, comma separated, and must be preceded by #. The tag function_name:<name_of_the_function> is automatically applied to custom metrics.

Note: The sum for each timestamp is used for counts and the last value for a given timestamp is used for gauges. It is not recommended to print a log statement every time you increment a metric, as this increases the time it takes to parse your logs. Continually update the value of the metric in your code, and print one log statement for that metric before the function finishes.