Network Performance Monitoring is now generally available! Network Monitoring is now available!

Amazon Lambda

Crawler Crawler

Overview

Amazon Lambda is a compute service that runs code in response to events and automatically manages the compute resources required by that code.

Enable this integration to begin collecting CloudWatch metrics. This page also describes how to set up custom metrics, logging, and tracing for your Lambda functions.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first.

Metric collection

  1. In the AWS integration tile, ensure that Lambda is checked under metric collection.

  2. Add the following permissions to your Datadog IAM policy to collect Amazon Lambda metrics. For more information on Lambda policies, review the documentation on the AWS website.

    AWS PermissionDescription
    lambda:List*List Lambda functions, metadata, and tags.
    tag:GetResourcesGet custom tags applied to Lambda functions.
  3. Install the Datadog - AWS Lambda integration.

Once this is completed, view all of your Lambda Functions in the Datadog Serverless view. This page brings together metrics, traces, and logs from your AWS Lambda functions running serverless applications into one view. Detailed documentation on this feature can be found in the Datadog Serverless documentation.

Tagging metrics

Any tag applied to your Lambda function automatically becomes a new dimension on which your can slice and dice your metrics.

Datadog Lambda Layer

The Datadog Lambda Layer is responsible for:

  • Submitting custom metrics (synchronously and asynchronously)
  • Automatically propagating tracing headers from upstream requests to downstream services. This enables full distributed tracing across Lambda functions, hosts, containers, and other infrastructure running the Datadog Agent.

Installing and using the Datadog Lambda Layer

Datadog offers Lambda Layers for Python, Node.js, and Ruby. Go is also supported with a package to include in your project. Datadog is working on support for new languages and runtimes; if there is another runtime you would like Datadog to support, reach out to the Datadog support team.

The Datadog Lambda Layer ARN includes a region, language runtime, and version. Construct yours in the following format:

  arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>:<VERSION>

For example:

  arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python37:8
LanguageRuntimeReleases
PythonPython27, Python36, Python37Latest release
Node.jsNode8-10, Node10-xLatest release
RubyRubyLatest release

Golang: Since Go binaries are statically-linked, Datadog offers a package you can simply import in your project. No Lambda Layer is required.

Note: The Datadog Lambda Layer and client libraries include the X-Ray SDK as a dependency, so you don’t need to explicitly install it in your projects.

You can configure the Datadog Lambda Layer by adding environment variables to your Lambda functions:

Environment VariableDescriptionRequiredDefaultAccepted Values
DD_API_KEYYour Datadog API keyYesDatadog API key
DD_KMS_API_KEYUse instead of DD_API_KEY if using KMSNoKMS-encrypted Datadog API key
DD_SITESet if using the EU instance of DatadogNodatadoghq.comdatadoghq.eu, datadoghq.com
DD_FLUSH_TO_LOGEnable zero latency asynchronous custom metricsNoFalseTrue, False
DD_LOG_LEVELEnable detailed logs for the Datadog Lambda LayerNoINFOINFO, DEBUG
  1. Navigate to the Lambda function to which you want to add the Layer in your AWS console.
  2. Click on Layers on the main page of your function.
  3. Scroll down, and click on Provide a layer version ARN.
  4. Enter the Datadog Lambda Layer ARN from the table above.
  5. Navigate to the Environment Variables section of your function to configure your Datadog API key, and any other options (see table above).

This plugin attaches the Datadog Lambda Layers for Node.js and Python to your functions. At deploy time, it generates new handler functions that wrap your existing functions and initializes the Lambda Layers.

You can install the plugin with one of the following commands.

npm install --save-dev serverless-plugin-datadog  # for NPM users
yarn add --dev serverless-plugin-datadog          # for Yarn users

Then, in your serverless.yml, add the following:

plugins:
  - serverless-plugin-datadog

Configure the library by adding the following section to your serverless.yml. The default values are listed, as well as whether the field is required.

custom:
  datadog:
    # (Required) Datadog API Key, only necessary when using synchronous custom metrics
    apiKey: "YOUR_DATADOG_API_KEY"

    # (Optional) Encrypted Datadog API Key (if using KMS)
    apiKMSKey: ""

    # (Optional) Controls if the plugin should add the Datadog Lambda Layers automatically
    #  Default: true
    #  Options: true, false
    addLayers: true

    # (Optional) Which Datadog instance to send data to
    #  Default: datadoghq.com
    #  Options: datadoghq.com, datadoghq.eu
    site: datadoghq.com

    # (Optional) Controls if asynchronous custom metric submission is enabled by default
    #  Default: false
    #  Options: true, false
    flushMetricsToLogs: false

    # (Optional) The log level of the Datadog plugin and Lambda Layers
    #  Default: "INFO"
    #  Options: INFO, DEBUG
    logLevel: "INFO"    

Serverless Framework docs

To enable X-Ray tracing by default for your Lambda functions and API Gateways, add the Function::Tracing and Api::TracingEnabled keys to the Globals section of your template.yaml. Also, add your Datadog API key and any other environment variables (see table above):

Globals:
  Function:
    Tracing: Active
    Environment:
      Variables:
        DD_API_KEY: YOUR_DATADOG_API_KEY
  Api:
    TracingEnabled: true

AWS SAM docs

Log collection

  1. Set up the Datadog log collection AWS Lambda function if you haven’t already.
  2. Once the Lambda function is installed, there are two ways to forward your Lambda logs to Datadog:

Once done, go to your Datadog Log section to start exploring your logs.

Trace collection

Datadog APM integrates with AWS X-Ray to collect your serverless traces and even connect them to Datadog traces from applications running on hosts and containers. Visualize your traces on the Serverless page, in App Analytics, and on the Service Map. Detailed documentation for setting up this integration can be found in the Datadog X-Ray integration page.

Tracing across AWS Lambda and hosts

When applicable, Datadog merges AWS X-Ray traces with native Datadog APM traces. This means that your traces will show the complete picture of requests that cross infrastructure boundaries, whether it be AWS Lambda, containers, on-prem hosts, or managed services.

  1. Enable the AWS X-Ray integration for tracing your Lambda functions.
  2. Add the Datadog Lambda Layer to your Lambda functions.
  3. Set up Datadog APM on your hosts and container-based infrastructure.

Note: For X-Ray and Datadog APM traces to appear in the same flame graph, all services must have the same env tag.

Organizing your infrastructure with tags

Any tag applied to your Lambda function automatically becomes a new dimension on which your can slice and dice your traces.

Tags are especially powerful in Datadog APM, the Service Map, and the Services List, which have first-class support for the env and service tags.

The env tag

Use env to separate out your staging, development, and production environments. This works for any kind of infrastructure, not just for your serverless functions. As an example, you could tag your production EU Lambda functions with env:prod-eu.

By default, Lambda functions are tagged with env:none in Datadog. Add your own tag to override this.

The service tag

Add the service tag in order to group related Lambda functions into a service. The Service Map and Services List use this tag to show relationships between services and the health of their monitors. Services are represented as individual nodes on the Service Map.

By default, each Lambda function is treated as its own service. Add your own tag to override this.

animated service map of Lambda functions

Custom metrics

Install the Datadog Lambda Layer to collect and send custom metrics. Metrics sent from the Datadog Lambda Layer are automatically aggregated into distributions, so you can graph the avg, sum, max, min, and count. You can also calculate aggregations over a set of tags for the 50th, 75th, 95th, and 99th percentile values on the Distribution Metrics page.

Distribution metrics are designed to instrument logical objects, like services, independent of the underlying hosts. So, they are well-suited for serverless infrastructure because they aggregate metrics server-side instead of locally with an Agent.

Upgrading to distribution metrics

With distribution metrics, you select the aggregation when graphing or querying it instead of specifying it at submission time.

If you previously submitted custom metrics from Lambda without using one of the Datadog Lambda Layers, you’ll need to start instrumenting your cuustom metrics under new metric names when submitting them to Datadog. The same metric name cannot simultaneously exist as both distribution and non-distribution metric type.

To enable percentile aggregations for your distribution metrics, consult the Distribution Metrics page.

Tagging custom metrics

You should tag your custom metrics when submitting them with the Datadog Lambda Layer. Use the Distribution Metrics page to customize the set of tags applied to your custom metrics.

Synchronous vs. asynchronous custom metrics

The Datadog Lambda Layer supports submitting custom metrics in Lambda both synchronously and asynchronously.

Synchronous: The default behavior. This method submits your custom metrics to Datadog via HTTP periodically (every 10 seconds) and at the end of your Lambda invocation. So if the invocation lasts for less than 10 seconds, your custom metrics will be submitted at the end of the invocation.

Asynchronous (recommended): It’s possible to submit your custom metrics with zero latency overhead and have them appear in Datadog in near-real time. To accomplish this, the Lambda Layer emits your custom metrics as a specially-formatted log line, which the Datadog Forwarder Lambda Function parses and submits to Datadog. Logging in AWS Lambda is 100% asynchronous, so this method ensures there is zero latency overhead to your function.

Enabling asynchronous custom metrics
  1. Set the environment variable DD_FLUSH_TO_LOG to True on your Lambda function.
  2. Update your Datadog Forwarder Lambda Function to at least version 1.4.0.

If you are not using Datadog Logs, you can still use asynchronous custom metric submission. Set the environment variable DD_FORWARD_LOG to False on the Datadog log collection AWS Lambda function. This intelligently forwards only custom metrics to Datadog, and not regular logs.

Custom metrics sample code

In your function code, you must import the necessary methods from the Lambda Layer and add a wrapper around your function handler.

Note: The arguments to the custom metrics reporting methods have the following requirements:

  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.

  • <METRIC_VALUE> MUST be a number (i.e. integer or float).

  • <TAG_LIST> is optional and formatted, for example: ['owner:Datadog', 'env:demo', 'cooltag'].

from datadog_lambda.metric import lambda_metric
from datadog_lambda.wrapper import datadog_lambda_wrapper


@datadog_lambda_wrapper
def lambda_handler(event, context):
    lambda_metric(
        "coffee_house.order_value",             # Metric name
        12.45,                                  # Metric value
        tags=['product:latte', 'order:online']  # Associated tags
    )
const { datadog, sendDistributionMetric } = require("datadog-lambda-js");

async function myHandler(event, context) {
  sendDistributionMetric(
    "coffee_house.order_value",       // Metric name
    12.45,                            // Metric value
    "product:latte", "order:online"   // Associated tags
  );
  return {
    statusCode: 200,
    body: "hello, dog!",
  };
}
// Wrap your handler function:
module.exports.myHandler = datadog(myHandler);

/* OR with manual configuration options
module.exports.myHandler = datadog(myHandler, {
    apiKey: "my-api-key"
});
*/
package main

import (
  "github.com/aws/aws-lambda-go/lambda"
  "github.com/DataDog/datadog-lambda-go"
)

func main() {
  // Wrap your handler function:
  lambda.Start(ddlambda.WrapHandler(myHandler, nil))
  /* OR with manual configuration options
  lambda.Start(ddlambda.WrapHandler(myHandler, &ddlambda.Config{
    BatchInterval: time.Second * 15
    APIKey: "my-api-key",
  }))
  */
}

func myHandler(ctx context.Context, event MyEvent) (string, error) {
  ddlambda.Distribution(
    "coffee_house.order_value",     // Metric name
    12.45,                          // Metric value
    "product:latte", "order:online" // Associated tags
  )
  // ...
}
require 'datadog/lambda'

def handler(event:, context:)
    # Wrap your handler function:
    Datadog::Lambda.wrap(event, context) do
        Datadog::Lambda.metric(
          'coffee_house.order_value',         # Metric name
          12.45,                              # Metric value
          "product":"latte", "order":"online" # Associated tags
        )
        return { statusCode: 200, body: 'Hello World' }
    end
end

Emitting asynchronous custom metrics is possible for any language or custom runtime. It works by printing a special JSON-formatted string in your Lambda function that the [Datadog Forwarder Lambda Function][8] identifies and submits to Datadog. To use this:

  1. Enable asynchronous custom metrics
  2. Write a reusable function that logs your custom metrics in the following format:
{
    "m": "Metric name",
    "v": "Metric value",
    "e": "Unix timestamp (seconds)",
    "t": "Array of tags"
}

For example:

{
    "m": "coffee_house.order_value",
    "v": 12.45,
    "e": 1572273854,
    "t": ["product:latte", "order:online"]
}

Note: These custom metrics are submitted as distributions. If you were previously submitting custom metrics another way, consult the documentation on the implications of upgrading to distributions.

Running in a VPC

The Datadog Lambda Layer requires access to the public internet in order to submit custom metrics synchronously. If your Lambda function is associated with a VPC, ensure that it is instead submitting custom metrics asynchronously or that your function can reach the public internet.

Using Third-Party Libraries

There are a number of open source libraries that make it easy to submit custom metrics to Datadog. However, many have not been updated to use Distribution metrics, which are optimized for Lambda. Distribution metrics allow for server-side aggregations independent of a host or locally-running agent. In a serverless environment where there is no agent, Distribution metrics give you flexible aggregations and tagging.

When evaluating third-party metrics libraries for AWS Lambda, ensure they support Distribution metrics.

[DEPRECATED] Using CloudWatch Logs

This method of submitting custom metrics is no longer supported, and is disabled for all new customers. The recommended way to submit custom metrics from Lambda is with a Datadog Lambda Layer.

This requires the following AWS permissions in your Datadog IAM policy.

AWS PermissionDescription
logs:DescribeLogGroupsList available log groups.
logs:DescribeLogStreamsList available log streams for a group.
logs:FilterLogEventsFetch specific log events for a stream to generate metrics.

[DEPRECATED] To send custom metrics to Datadog from your Lambda logs, print a log line using the following format:

MONITORING|<UNIX_EPOCH_TIMESTAMP>|<METRIC_VALUE>|<METRIC_TYPE>|<METRIC_NAME>|#<TAG_LIST>

Where:

  • MONITORING signals to the Datadog integration that it should collect this log entry.

  • <UNIX_EPOCH_TIMESTAMP> is in seconds, not milliseconds.

  • <METRIC_VALUE> MUST be a number (i.e. integer or float).

  • <METRIC_TYPE> is count, gauge, histogram, or check.

  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.

  • <TAG_LIST> is optional, comma separated, and must be preceded by #. The tag function_name:<name_of_the_function> is automatically applied to custom metrics.

Note: The sum for each timestamp is used for counts and the last value for a given timestamp is used for gauges. It is not recommended to print a log statement every time you increment a metric, as this increases the time it takes to parse your logs. Continually update the value of the metric in your code, and print one log statement for that metric before the function finishes.

Data Collected

Metrics

aws.lambda.duration
(gauge)
Measures the average elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.maximum
(gauge)
Measures the maximum elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.minimum
(gauge)
Measures the minimum elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.sum
(gauge)
Measures the total execution time of the lambda function executing.
Shown as millisecond
aws.lambda.duration.p80
(gauge)
Measures the p80 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.p95
(gauge)
Measures the p95 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.p99
(gauge)
Measures the p99 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.p99.9
(gauge)
Measures the p99.9 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.timeout
(gauge)
Measures the amount of allowed execution time for the function before the Lambda runtime stops it.
Shown as second
aws.lambda.errors
(count)
Measures the number of invocations that failed due to errors in the function (response code 4XX).
Shown as error
aws.lambda.invocations
(count)
Measures the number of times a function is invoked in response to an event or invocation API call.
Shown as invocation
aws.lambda.throttles
(count)
Measures the number of Lambda function invocation attempts that were throttled due to invocation rates exceeding the customer's concurrent limits (error code 429). Failed invocations may trigger a retry attempt that succeeds.
Shown as throttle
aws.lambda.iterator_age
(gauge)
Measures the age of the last record for each batch of records processed
Shown as millisecond
aws.lambda.iterator_age.minimum
(gauge)
Measures the minimum age of the last record for each batch of records processed
Shown as millisecond
aws.lambda.iterator_age.maximum
(gauge)
Measures the maximum age of the last record for each batch of records processed
Shown as millisecond
aws.lambda.iterator_age.sum
(gauge)
Measures the sum of the ages of the last record for each batch of records processed
Shown as millisecond
aws.lambda.dead_letter_errors
(count)
Measures the sum of times Lambda is unable to write the failed event payload to your configured Dead Letter Queues.
Shown as error
aws.lambda.concurrent_executions
(gauge)
Measures the average of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.concurrent_executions.minimum
(gauge)
Measures the minimum of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.concurrent_executions.maximum
(gauge)
Measures the maximum of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.concurrent_executions.sum
(gauge)
Measures the sum of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.unreserved_concurrent_executions
(gauge)
Measures the sum of the concurrency of the functions that don't have a custom concurrency limit specified.
Shown as execution
aws.lambda.provisioned_concurrent_executions
(count)
Measures the sum of the provisioned concurrent executions for a given function
Shown as execution
aws.lambda.provisioned_concurrency_invocations
(count)
Measures the sum of the concurrent executions using the provisioned concurrency
Shown as invocation
aws.lambda.provisioned_concurrency_spillover_invocations
(count)
Measures the sum of the concurrent executions over the provisioned concurrency
Shown as invocation
aws.lambda.provisioned_concurrency_utilization
(gauge)
Measures the average fraction of provisioned concurrency in use for a given function at a given point in time
Shown as percent
aws.lambda.provisioned_concurrency_utilization.minimum
(gauge)
Measures the minimum fraction of provisioned concurrency in use for a given function at a given point in time
Shown as percent
aws.lambda.provisioned_concurrency_utilization.maximum
(gauge)
Measures the maximum fraction of provisioned concurrency in use for a given function at a given point in time
Shown as percent

Each of the metrics retrieved from AWS is assigned the same tags that appear in the AWS console, including but not limited to function name, security-groups, and more.

Custom metrics are only tagged with function name.

Events

The AWS Lambda integration does not include any events.

Service Checks

The AWS Lambda integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.

Further Reading