Datadog Lambda Extension

Datadog Lambda Extension

Overview

AWS Lambda extensions are companion processes that augment your Lambda functions. They run within the Lambda execution environment, alongside your Lambda function code. The Datadog extension is a lightweight version of the Datadog Agent built to run alongside your code with minimal performance overhead.

The Datadog Lambda Extension is responsible for:

The Datadog extension submits custom metrics, enhanced metrics, traces, and logs asynchronously. Submitting Lambda logs with the extension is supported in all Lambda runtimes. Submitting custom metrics, enhanced metrics and traces is supported in Node.js, Python, and Go Lambda runtimes.

Installation

To install the Datadog Lambda Extension, instrument your AWS serverless applications, and review supported runtimes, see the serverless installation instructions.

Log collection

To disable submission of your AWS Lambda logs to Datadog using the extension, set the environment variable DD_SERVERLESS_LOGS_ENABLED to false in your Lambda function.

To exclude the START and END logs, set the environment variable DD_LOGS_CONFIG_PROCESSING_RULES to [{"type": "exclude_at_match", "name": "exclude_start_and_end_logs", "pattern": "(START|END)\\s"}]. Alternatively, you can add a datadog.yaml file in your project root directory with the following content:

logs_config:
  processing_rules:
    - type: exclude_at_match
      name: exclude_start_and_end_logs
      pattern: (START|END)\s

To scrub or filter other logs before sending them to Datadog, see Advanced Log Collection.

Trace collection

To disable submission of your AWS Lambda traces to Datadog using the extension, set the environment variable DD_TRACE_ENABLED to false in your Lambda function.

To filter traces before sending them to Datadog, see Ignoring Unwanted Resources in APM.

To scrub trace attributes for data security, see Configure the Datadog Agent or Tracer for Data Security.

Tagging

When using the Extension, Datadog recommends you apply tags by adding the following environment variables to your Lambda functions:

  • DD_ENV: This sets the env tag in Datadog. Use it to separate out your staging, development, and production environments.
  • DD_SERVICE: This sets the service tag in Datadog. Use it to group related Lambda functions into a service.
  • DD_VERSION: This sets the version tag in Datadog. Use it to enable Deployment Tracking.
  • DD_TAGS: This sets custom tags in Datadog. Must be a list of <key>:<value> separated by commas such as: layer:api,team:intake.

If you have the Datadog AWS integration enabled, any AWS resource tag applied to your AWS Lambda function also gets applied in Datadog automatically.

VPC

The Datadog Lambda Extension needs access to public internet to send data to Datadog. If your Lambda functions are deployed in a VPC private subnet without access to public internet, see the options below.

If you are sending data to a Datadog site that is hosted on AWS, such as US1, then you can send data over AWS PrivateLink. See Datadog Sites or file a ticket at help.datadoghq.com if you are unsure about your Datadog site.

Using a proxy

If you are sending data to a Datadog site that is NOT hosted on AWS, such as EU1, then you need to send data over a proxy.

Overhead

The Datadog Lambda Extension introduces a small amount of overhead to your Lambda function’s cold starts (that is, the higher init duration), as the Extension needs to initialize. Datadog is continuously optimizing the Lambda extension performance and recommend always using the latest release.

You may notice an increase of your Lambda function’s reported duration. This is because the Datadog Lambda Extension needs to flush data back to the Datadog API. Although the time spent by the extension flushing data is reported as part of the duration, it’s done after AWS returns your function’s response back to the client. In other words, the added duration does not slow down your Lambda function. See this AWS blog post for more technical information.

By default, the Extension sends data back to Datadog at the end of each invocation. This avoids delays of data arrival for sporadic invocations from low-traffic applications, cron jobs, and manual tests. Once the Extension detects a steady and frequent invocation pattern (more than once per minute), it batches data from multiple invocations and flushes periodically at the beginning of the invocation when it’s due. This means that the busier your function is, the lower the average duration overhead per invocation.

For Lambda functions deployed in a region that is far from the Datadog site, for example, a Lambda function deployed in eu-west-1 reporting data to the US1 Datadog site, can observe a higher duration overhead due to the network latency. You can set the environment variable DD_SERVERLESS_FLUSH_STRATEGY with value periodically,30000 on your Lambda functions to flush data every 30s, instead of the default every 10s, and this usually results in a significantly lower average duration overhead per invocation.

Further Reading