New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Amazon Lambda

Crawler Crawler

Overview

Amazon Lambda is a compute service that runs code in response to events and automatically manages the compute resources required by that code.

Enable this integration to begin collecting Cloudwatch and custom metrics from your Lambda functions.

Datadog supports the following features depending of your AWS Lambda language:

PythonNodeJavaGoRuby.NET & Core
Lambda Runtime Metrics
Tracing
Custom MetricsLambda LayerLambda LayerSent via LogsGo PackageLambda LayerSent via Logs
Logging

Host-agnostic end-to-end distributed tracing allows you to trace requests that span the entirety of your infrastructure, both on Lambda and on hosts running the Datadog Agent. Essentially, X-Ray traces are stitched to Datadog APM traces, enabling you to access a holistic view of your system.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first.

Metric collection

  1. In the AWS integration tile, ensure that Lambda is checked under metric collection.

  2. Add the following permissions to your Datadog IAM policy to collect Amazon Lambda metrics. For more information on Lambda policies, review the documentation on the AWS website.

    AWS PermissionDescription
    lambda:List*List Lambda functions, metadata, and tags.
    logs:DescribeLogGroupsList available log groups.
    logs:DescribeLogStreamsList available log streams for a group.
    logs:FilterLogEventsFetch specific log events for a stream to generate metrics.
    tag:GetResourcesGet custom tags applied to Lambda functions.
  3. Install the Datadog - AWS Lambda integration.

Once this is completed, view all of your Lambda Functions in the Datadog Serverless UI. This UI brings together metrics, traces, and logs from your AWS Lambda functions running serverless applications into one view. Detailed documentation on this feature can be found in the Datadog Serverless documentation.

Log collection

  1. Set up the Datadog log collection AWS Lambda function if you haven’t already.
  2. Once the Lambda function is installed, there are two ways to forward your Lambda logs to Datadog:

Once done, go to your Datadog Log section to start exploring your logs.

Trace collection

Datadog integrates with AWS X-Ray to collect and visualize your serverless traces from Lambda based applications in the Serverless UI. Detailed documentation for setting up this integration can be found in the Datadog Serverless documentation.

Custom metrics

If your Lambda function is using a Python runtime, install the Datadog Lambda Layer to collect and send custom metrics. Metrics sent from the Datadog Lambda Layer are automatically aggregated into distributions, so you can graph the average, sum, max, min, and count, as well as 50th, 75th, 95th, and 99th percentile values. If you are using another runtime, log your custom metrics in a specific format to be picked up and parsed into metrics by Datadog.

Installing and using the Datadog Layer

Datadog’s Lambda Layer supports Python, Node, and Ruby. Go is also supported with a package to include in your project. If there is another runtime you would like Datadog to support, reach out to the Datadog support team.

  1. Navigate to the Lambda function to which you want to add the Layer in your AWS console.
  2. Click on Layers on the main page of your function.
  3. Scroll down, and click on Provide a layer version ARN.
  4. Enter an ARN in the following format, providing your desired region, runtime, and the latest Lambda Layer version that can be found the table below:

    arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>:<VERSION>
    

    For example:

    arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python37:3
    
    LanguageRuntimeReleases
    PythonPython27, Python36, Python37Latest release
    NodeNode8-10, Node10-xLatest release
    RubyRubyLatest release

    Go users: Because Go is statically-compiled, you must include the package in your project instead of using a Lambda Layer.

  5. Navigate to the Environment Variables section of your function and set the following environment variable with your Datadog API key:

    • DD_API_KEY or DD_KMS_API_KEY (if encrypted by KMS)
  6. For completely asynchronous, near-real time custom metrics, optionally set the environment variable DD_FLUSH_TO_LOG to True. This will submit metrics through CloudWatch Logs instead of API calls. Note: This option requires setting up the latest version of the Datadog log collection AWS Lambda function. If you only want to forward metrics from CloudWatch Logs, you MUST set the environment variable DD_FORWARD_LOG to False on the Datadog log collection AWS Lambda function, to avoid forwarding regular logs to Datadog.

In your function code, you must import the necessary methods from the layer and add a wrapper around your function handler.

from datadog_lambda.metric import lambda_metric
from datadog_lambda.wrapper import datadog_lambda_wrapper


@datadog_lambda_wrapper
def lambda_handler(event, context):
    lambda_metric(
        "coffee_house.order_value",             # Metric name
        12.45,                                  # Metric value
        tags=['product:latte', 'order:online']  # Associated tags
    )
const { datadog, sendDistributionMetric } = require("datadog-lambda-js");

async function myHandler(event, context) {
  sendDistributionMetric(
    "coffee_house.order_value",       // Metric name
    12.45,                            // Metric value
    "product:latte", "order:online"   // Associated tags
  );
  return {
    statusCode: 200,
    body: "hello, dog!",
  };
}
// Wrap your handler function:
module.exports.myHandler = datadog(myHandler);

/* OR with manual configuration options
module.exports.myHandler = datadog(myHandler, {
    apiKey: "my-api-key"
});
*/
package main

import (
  "github.com/aws/aws-lambda-go/lambda"
  "github.com/DataDog/datadog-lambda-go"
)

func main() {
  // Wrap your handler function:
  lambda.Start(ddlambda.WrapHandler(myHandler, nil))
  /* OR with manual configuration options
  lambda.Start(ddlambda.WrapHandler(myHandler, &ddlambda.Config{
    BatchInterval: time.Second * 15
    APIKey: "my-api-key",
  }))
  */
}

func myHandler(ctx context.Context, event MyEvent) (string, error) {
  ddlambda.Distribution(
    "coffee_house.order_value",     // Metric name
    12.45,                          // Metric value
    "product:latte", "order:online" // Associated tags
  )
  // ...
}
require 'datadog/lambda'

def handler(event:, context:)
    # Wrap your handler function:
    Datadog::Lambda.wrap(event, context) do
        Datadog::Lambda.metric(
          'coffee_house.order_value',         # Metric name
          12.45,                              # Metric value
          "product":"latte", "order":"online" # Associated tags
        )
        return { statusCode: 200, body: 'Hello World' }
    end
end

The arguments to the custom metrics reporting methods have the following requirements:

lambda_metric(<METRIC_NAME>, <METRIC_VALUE>, tags=[<TAG_LIST>])

Where:

  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.

  • <METRIC_VALUE> MUST be a number (i.e. integer or float).

  • <TAG_LIST> is optional and formatted, for example: ['owner:Datadog', 'env:demo', 'cooltag'].

Note: There is currently a max of 3 tags that can be applied to a metric. If you need more than 3 tags on these metrics, contact the Datadog support team to have this limit raised for your organization.

Detailed instructions for installing and configuring Lambda Layers can be found in the AWS main documentation.

Running in a VPC

If your Lambda function is associated with a VPC and sending custom metrics synchronously, you need to ensure it has access to the public internet.

Using CloudWatch Logs

To send custom metrics to Datadog from your Lambda logs, print a log line from your Lambda, using the following format:

MONITORING|<UNIX_EPOCH_TIMESTAMP>|<METRIC_VALUE>|<METRIC_TYPE>|<METRIC_NAME>|#<TAG_LIST>

Where:

  • MONITORING signals to the Datadog integration that it should collect this log entry.

  • <UNIX_EPOCH_TIMESTAMP> is in seconds, not milliseconds.

  • <METRIC_VALUE> MUST be a number (i.e. integer or float).

  • <METRIC_TYPE> is count, gauge, histogram, or check.

  • <METRIC_NAME> uniquely identifies your metric and adheres to the metric naming policy.

  • <TAG_LIST> is optional, comma separated, and must be preceded by #. The tag function_name:<name_of_the_function> is automatically applied to custom metrics.

Note: The sum for each timestamp is used for counts and the last value for a given timestamp is used for gauges. It is not recommended to print a log statement every time you increment a metric, as this increases the time it takes to parse your logs. Continually update the value of the metric in your code, and print one log statement for that metric before the function finishes.

The Lambda Layer does not work in a VPC unless it has access to the public internet. If you are using a VPC, log your custom metrics to CloudWatch Logs so that Datadog can parse them.

Data Collected

Metrics

aws.lambda.duration
(gauge)
Measures the average elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.maximum
(gauge)
Measures the maximum elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.minimum
(gauge)
Measures the minimum elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.sum
(gauge)
Measures the total execution time of the lambda function executing.
Shown as millisecond
aws.lambda.duration.p80
(gauge)
Measures the p80 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.p95
(gauge)
Measures the p95 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.p99
(gauge)
Measures the p99 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.duration.p99.9
(gauge)
Measures the p99.9 elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing.
Shown as millisecond
aws.lambda.timeout
(gauge)
Measures the amount of allowed execution time for the function before the Lambda runtime stops it.
Shown as second
aws.lambda.errors
(count)
Measures the number of invocations that failed due to errors in the function (response code 4XX).
Shown as error
aws.lambda.invocations
(count)
Measures the number of times a function is invoked in response to an event or invocation API call.
Shown as invocation
aws.lambda.throttles
(count)
Measures the number of Lambda function invocation attempts that were throttled due to invocation rates exceeding the customer's concurrent limits (error code 429). Failed invocations may trigger a retry attempt that succeeds.
Shown as throttle
aws.lambda.iterator_age
(gauge)
Measures the age of the last record for each batch of records processed
Shown as millisecond
aws.lambda.iterator_age.minimum
(gauge)
Measures the minimum age of the last record for each batch of records processed
Shown as millisecond
aws.lambda.iterator_age.maximum
(gauge)
Measures the maximum age of the last record for each batch of records processed
Shown as millisecond
aws.lambda.iterator_age.sum
(gauge)
Measures the sum of the ages of the last record for each batch of records processed
Shown as millisecond
aws.lambda.dead_letter_errors
(count)
Measures the sum of times Lambda is unable to write the failed event payload to your configured Dead Letter Queues.
Shown as error
aws.lambda.concurrent_executions
(gauge)
Measures the average of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.concurrent_executions.minimum
(gauge)
Measures the minimum of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.concurrent_executions.maximum
(gauge)
Measures the maximum of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.concurrent_executions.sum
(gauge)
Measures the sum of concurrent executions for a given function at a given point in time.
Shown as execution
aws.lambda.unreserved_concurrent_executions
(gauge)
Measures the sum of the concurrency of the functions that don't have a custom concurrency limit specified.
Shown as execution

Each of the metrics retrieved from AWS is assigned the same tags that appear in the AWS console, including but not limited to function name, security-groups, and more.

Custom metrics are only tagged with function name.

Events

The AWS Lambda integration does not include any events.

Service Checks

The AWS Lambda integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.

Further Reading