By connecting your serverless traces to metrics, Datadog provides a context-rich picture of your application’s performance, allowing you to better troubleshoot performance issues given the distributed nature of serverless applications.
The Datadog Python, Node.js, Ruby, Go, Java, and .NET tracing libraries support distributed tracing for AWS Lambda.
To start using Datadog APM with your serverless application, you can choose between generating traces using Datadog’s tracing client (dd-trace) or pulling X-Ray traces from AWS.
| Datadog APM with dd-trace | Datadog APM with AWS X-Ray |
|---|---|
| Uses Datadog APM’s integration libraries for end-to-end tracing. | Pulls traces from AWS X-Ray. |
| Visualize your traces in Datadog in real-time. | Trace data available in Datadog after a few minutes. |
| Tail-based sampling and fully customizable tag-based retention filters. | Sampling rate cannot be configured. |
| Support for all Lambda runtimes. | Support for all Lambda runtimes. |
The Datadog Lambda Library and tracing libraries for Python and Node.js support:
For Python and Node.js serverless applications, Datadog recommends you install Datadog’s tracing libraries. If your application requires AWS X-Ray active tracing in AWS managed services such as AppSync or Step Functions, Datadog recommends you augment AWS X-Ray traces with Datadog APM by configuring both AWS X-Ray and Datadog APM tracing libraries as described in Serverless Trace Merging.
If you are already tracing your serverless functions with X-Ray and want to continue using X-Ray, you can install the AWS X-Ray integration.
Looking to trace through serverless resources not listed above? Open a feature request.
The Datadog Lambda Library and tracing libraries for Ruby support:
You can trace your serverless functions in Datadog with Datadog’s tracing libraries or by installing the AWS X-Ray integration. If you are using Datadog’s tracing libraries, and need to connect Lambda function traces across AWS managed services, Datadog recommends you augment your traces by configuring both AWS X-Ray and Datadog APM tracing libraries.
Looking to trace through serverless resources not listed above? Open a feature request.
The Datadog Lambda Library and tracing libraries for Go support:
For Go serverless applications, Datadog recommends installing Datadog’s tracing libraries. If your application requires AWS X-Ray active tracing in AWS managed services such as API Gateway or Step Functions, you may want to consider instead using Datadog APM with AWS X-Ray tracing.
Looking to trace through serverless resources not listed above? Open a feature request.
The Datadog Lambda Library and tracing libraries for Java support:
For Java serverless applications, Datadog recommends installing Datadog’s tracing libraries. If your application requires AWS X-Ray active tracing in AWS managed services such as API Gateway or Step Functions, you may want to consider instead using Datadog APM with AWS X-Ray tracing.
Have feedback on the Datadog’s tracing libraries for Java Lambda functions? Make sure to check out discussions going on in the #serverless channel in the Datadog Slack community.
The tracing library for .NET supports:
For .NET serverless applications, Datadog recommends installing Datadog’s tracing libraries. If your application requires AWS X-Ray active tracing in AWS managed services such as API Gateway or Step Functions, you may want to consider instead using Datadog APM with AWS X-Ray tracing.
Learn more about tracing through .NET Azure serverless applications.
If you have installed Datadog’s tracing libraries (dd-trace) on both your Lambda functions and hosts, your traces automatically show you the complete picture of requests that cross infrastructure boundaries, whether it be AWS Lambda, containers, on-prem hosts, or managed services.
If dd-trace is installed on your hosts with the Datadog Agent, and your serverless functions are traced with AWS X-Ray, trace merging is required to see a single, connected trace across your infrastructure. See the Serverless Trace Merging documentation to learn more about merging traces from dd-trace and AWS X-Ray.
Datadog’s AWS X-Ray integration only provides traces for Lambda functions. See the Datadog APM documentation to learn more about tracing in container or host-based environments.
The Datadog Python, Node.js, Ruby, Go, Java, and .NET tracing libraries support distributed tracing for AWS Lambda. To enable tracing on your functions, follow the installation instructions.
Datadog recommends using only the Datadog APM trace library (dd-trace), but in some advanced situations users can combine Datadog tracing and AWS X-Ray using trace merging. Trace merging is available for Node.js and Python AWS Lambda functions. If you aren’t sure which tracing library to use, read about choosing your tracing library.
There are two primary reasons for instrumenting both dd-trace and AWS X-Ray tracing libraries:
dd-trace, you require AWS X-Ray active tracing for AWS managed services such as AppSync and Step Functions, and you want to visualize the dd-trace and AWS X-Ray spans in one single trace.dd-trace instruments your hosts, AWS X-Ray instruments your Lambda functions, and you want to visualize connected traces for transactions across Lambda functions and hosts.Note: This may result in higher usage bills. X-Ray spans continue to be available in your merged traces after 2-5 minutes. In many cases, Datadog recommends only using a single tracing library. Learn more about choosing your tracing library.
You can find setup instructions for each of the above use cases below:
AWS X-Ray provides both a backend AWS service (AWS X-Ray active tracing) and a set of client libraries. Enabling the backend AWS service alone in the Lambda console gives you Initialization and Invocation spans for your AWS Lambda functions. You can also enable AWS X-Ray active tracing from the API Gateway and Step Function consoles.
Both the AWS X-Ray SDK and Datadog APM client libraries (dd-trace) add metadata and spans for downstream calls by accessing the function directly. Assuming you are using dd-trace to trace at the handler level, your setup should be similar to the following:
dd-trace) by following the installation instructions for your Lambda runtime.dd-trace, so the AWS X-Ray client libraries do not need to be installed.DD_MERGE_XRAY_TRACES environment variable to true on your Lambda functions to merge the X-Ray and dd-trace traces (DD_MERGE_DATADOG_XRAY_TRACES in Ruby).If you have installed Datadog’s tracing libraries (dd-trace) on both your Lambda functions and hosts, your traces will automatically show you the complete picture of requests that cross infrastructure boundaries, whether it be AWS Lambda, containers, on-prem hosts, or managed services.
If dd-trace is installed on your hosts with the Datadog Agent, and your Node.js or Python serverless functions are traced with AWS X-Ray, your setup should be similar to the following:
DD_TRACE_ENABLED environment variable is set to false.Then, for X-Ray and Datadog APM traces to appear in the same flame graph, all services must have the same env tag.
Note: Distributed Tracing is supported for any runtime for your host or container-based applications. Your hosts and Lambda functions do not need to be in the same runtime.
Additional instrumentation is sometimes required to see a single, connected trace in Node and Python serverless applications asynchronously triggering Lambda functions. If you are just getting started with monitoring serverless applications in Datadog, follow our main installation steps and read this page on choosing your tracing library. Once you are sending traces from your Lambda functions to Datadog using the Datadog Lambda Library, you may want to follow these steps to connect traces between two Lambda functions in cases such as:
Tracing many AWS Managed services (listed here) is supported out-of-the-box and does not require following the steps outlined on this page.
To successfully connect trace context between resources sending traces, you need to:
dd-trace installed.The following code samples outline how to pass trace context in outgoing payloads to services which do not support HTTP headers, or managed services not supported natively by Datadog in Node and Python:
In Python, you can use the get_dd_trace_context helper function to pass tracing context to outgoing events in a Lambda functions:
import json
import boto3
import os
from datadog_lambda.tracing import get_dd_trace_context # Datadog tracing helper function
def handler(event, context):
my_custom_client.sendRequest(
{
'myCustom': 'data',
'_datadog': {
'DataType': 'String',
'StringValue': json.dumps(get_dd_trace_context()) # Includes trace context in outgoing payload.
},
},
)
In Node, you can use the getTraceHeaders helper function to pass tracing context to outgoing events in a Lambda function:
const { getTraceHeaders } = require("datadog-lambda-js"); // Datadog tracing helper function
module.exports.handler = async event => {
const _datadog = getTraceHeaders(); // Captures current Datadog trace context.
var payload = JSON.stringify({ data: 'sns', _datadog });
await myCustomClient.sendRequest(payload)
If you aren’t passing trace context from your Lambda functions, you can use the following code template in place of the getTraceHeaders and get_dd_trace_context helper functions to get the current span context. Instructions on how to do this in every runtime are outlined here.
const tracer = require("dd-trace");
exports.handler = async event => {
const span = tracer.scope().active();
const _datadog = {}
tracer.inject(span, 'text_map', _datadog)
// ...
To extract the above trace context from the consumer Lambda function, you need to define an extractor function that captures trace context before the execution of your Lambda function handler. To do this, configure the DD_TRACE_EXTRACTOR environment variable to point to the location of your extractor function. The format for this is <FILE NAME>.<FUNCTION NAME>. For example, extractors.json if the json extractor is in the extractors.js file. Datadog recommends you place your extractor methods all in one file, as extractors can be re-used across multiple Lambda functions. These extractors are completely customizable to fit any use case.
Notes:
import or require your Node.js module where the extractors are defined. This ensures the module gets compiled and bundled into your Lambda deployment package.arm64, you must define the extractor in your function code instead of using the DD_TRACE_EXTRACTOR environment variable.The following code samples outline sample extractors you might use for propagating trace context across a third party system, or an API which does not support standard HTTP headers.
def extractor(payload):
trace_headers = json.loads(payload["_datadog"]);
trace_id = trace_headers["x-datadog-trace-id"];
parent_id = trace_headers["x-datadog-parent-id"];
sampling_priority = trace_headers["x-datadog-sampling-priority"];
return trace_id, parent_id, sampling_priority
exports.json = (payload) => {
const traceData = payload._datadog
const traceID = traceData["x-datadog-trace-id"];
const parentID = traceData["x-datadog-parent-id"];
const sampledHeader = traceData["x-datadog-sampling-priority"];
const sampleMode = parseInt(sampledHeader, 10);
return {
parentID,
sampleMode,
source: 'event',
traceID,
};
};
var exampleSQSExtractor = func(ctx context.Context, ev json.RawMessage) map[string]string {
eh := events.SQSEvent{}
headers := map[string]string{}
if err := json.Unmarshal(ev, &eh); err != nil {
return headers
}
// Using SQS as a trigger with a batchSize=1 so it's important we check
// for this as a single SQS message will drive the execution of the handler.
if len(eh.Records) != 1 {
return headers
}
record := eh.Records[0]
lowercaseHeaders := map[string]string{}
for k, v := range record.MessageAttributes {
if v.StringValue != nil {
lowercaseHeaders[strings.ToLower(k)] = *v.StringValue
}
}
return lowercaseHeaders
}
cfg := &ddlambda.Config{
TraceContextExtractor: exampleSQSExtractor,
}
ddlambda.WrapFunction(handler, cfg)
Additional helpful documentation, links, and articles:
