--- title: Distributed Tracing with AWS Lambda Serverless Applications description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Serverless > Serverless Monitoring for AWS Lambda > Distributed Tracing with AWS Lambda Serverless Applications --- # Distributed Tracing with AWS Lambda Serverless Applications {% image source="https://docs.dd-static.net/images/tracing/serverless_functions/ServerlessDistributedTrace.8849d3898d8b3c33b995508247c9dc7c.png?auto=format" alt="Trace Serverless Functions" /%} By connecting your serverless traces to metrics, Datadog provides a context-rich picture of your application's performance, allowing you to better troubleshoot performance issues given the distributed nature of serverless applications. The Datadog Python, Node.js, Ruby, Go, Java, and .NET tracing libraries support distributed tracing for AWS Lambda. ## Send traces from your serverless application{% #send-traces-from-your-serverless-application %} {% image source="https://docs.dd-static.net/images/serverless/serverless_tracing_installation_instructions.916e542d516b6e0e19d5f1dcf45dc027.png?auto=format" alt="Architecture diagram for tracing AWS Lambda with Datadog" /%} The Datadog Python, Node.js, Ruby, Go, Java, and .NET tracing libraries support distributed tracing for AWS Lambda. You can install the tracer using the [installation instructions](https://docs.datadoghq.com/serverless/installation/). ### Runtime recommendations{% #runtime-recommendations %} - [Python](https://docs.datadoghq.com/serverless/distributed_tracing#python-and-nodejs) - [Node.js](https://docs.datadoghq.com/serverless/distributed_tracing#python-and-nodejs) - [Ruby](https://docs.datadoghq.com/serverless/distributed_tracing#ruby) - [Java](https://docs.datadoghq.com/serverless/distributed_tracing#java) - [go](https://docs.datadoghq.com/serverless/distributed_tracing#go) - [.NET](https://docs.datadoghq.com/serverless/distributed_tracing#net) #### Python and Node.js{% #python-and-nodejs %} The Datadog Lambda Library and tracing libraries for Python and Node.js support: - Automatic correlation of Lambda logs and traces with trace ID and tag injection. - Installation without any code changes using Serverless Framework, AWS SAM and AWS CDK integrations. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing consecutive Lambda invocations made via the AWS SDK. - Cold start tracing - Tracing asynchronous Lambda invocations through AWS Managed Services - API Gateway - SQS - SNS - SNS and SQS direct integration - Kinesis - EventBridge - DynamoDB - S3 - Step Functions - Tracing dozens of additional out-of-the-box [Python](https://docs.datadoghq.com/tracing/trace_collection/compatibility/python) and [Node.js](https://docs.datadoghq.com/tracing/trace_collection/compatibility/nodejs) libraries. For Python and Node.js serverless applications, Datadog recommends you [install Datadog's tracing libraries](https://docs.datadoghq.com/serverless/installation/). *Looking to trace through serverless resources not listed above? [Open a feature request](https://docs.datadoghq.com/help/).* #### Ruby{% #ruby %} The Datadog Lambda Library and tracing libraries for Ruby support: - Automatic correlation of Lambda logs and traces with trace ID and tag injection. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [Ruby](https://docs.datadoghq.com/tracing/trace_collection/compatibility/ruby) libraries. You can trace your serverless functions in Datadog with [Datadog's tracing libraries](https://docs.datadoghq.com/serverless/installation/). *Looking to trace through serverless resources not listed above? [Open a feature request](https://docs.datadoghq.com/help/).* #### Go{% #go %} The Datadog Lambda Library and tracing libraries for Go support: - Manual correlation of Lambda logs and traces with trace ID and tag injection. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [Go](https://docs.datadoghq.com/tracing/trace_collection/compatibility/go) libraries. For Go serverless applications, Datadog recommends installing [Datadog's tracing libraries](https://docs.datadoghq.com/serverless/installation/). *Looking to trace through serverless resources not listed above? [Open a feature request](https://docs.datadoghq.com/help/).* #### Java{% #java %} The Datadog Lambda Library and tracing libraries for Java support: - Correlation of Lambda logs and traces with trace ID and tag injection. See [Connecting Java logs and traces](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/java/) for more details. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [Java](https://docs.datadoghq.com/tracing/trace_collection/compatibility/java) libraries. For Java serverless applications, Datadog recommends [installing Datadog's tracing libraries](https://docs.datadoghq.com/serverless/installation/). *Have feedback on the Datadog's tracing libraries for Java Lambda functions? Make sure to check out discussions going on in the [\#serverless](https://datadoghq.slack.com/archives/CFDPB83M4) channel in the [Datadog Slack community](https://chat.datadoghq.com/).* #### .NET{% #net %} The tracing library for .NET supports: - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [.NET](https://docs.datadoghq.com/tracing/trace_collection/compatibility/dotnet-core) libraries. For .NET serverless applications, Datadog recommends [installing Datadog's tracing libraries](https://docs.datadoghq.com/serverless/installation/). Learn more about [tracing through .NET Azure serverless applications](https://docs.datadoghq.com/serverless/azure_app_services). ## Span Auto-linking{% #span-auto-linking %} {% image source="https://docs.dd-static.net/images/serverless/lambda/tracing/autolink.fa059aab20b6ec455211424c2d24a0ef.png?auto=format" alt="In Datadog, a DynamoDB trace. At the top, a message reads 'This trace is linked to other traces'. The Span Links tab is open and displays a clickable link to another DynamoDB trace." /%} Datadog automatically detects linked spans when segments of your asynchronous requests cannot propagate trace context. For example, this may occur when a request triggers an [S3 Change Events](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), or [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html). You can see Auto-linked spans appear in the [Span Links tab](https://docs.datadoghq.com/tracing/trace_explorer/trace_view/?tab=spanlinksbeta). These appear as **Backward** or **Forward**. *Backward*: The linked span was caused by the trace you are viewing. *Forward*: The linked span caused the trace you are viewing. {% alert level="info" %} Sampling and [trace retention filters](https://docs.datadoghq.com/tracing/trace_pipeline/trace_retention/) can interfere with Auto-linking. To improve your chances of seeing Auto-linked spans, increase your sample rate or adjust your retention filters. {% /alert %} ### Supported technologies{% #supported-technologies %} Span Auto-linking is available for: - Python AWS Lambda functions instrumented with [`datadog-lambda-python`](https://github.com/DataDog/datadog-lambda-python) layer v101+ - Python applications instrumented with [`dd-trace-py`](https://github.com/DataDog/dd-trace-py/) v2.16+ - Node.js AWS Lambda functions instrumented with [`datadog-lambda-js`](https://github.com/DataDog/datadog-lambda-js) layer 118+ - Node.js applications instrumented with [`dd-trace-js`](https://github.com/DataDog/dd-trace-js/) v4.53.0+ or v5.29.0+ ### DynamoDB Change Stream Auto-linking{% #dynamodb-change-stream-auto-linking %} For [DynamoDB Change Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html), Span Auto-linking supports the following operations: - `PutItem` - `UpdateItem` - `DeleteItem` - `BatchWriteItem` - `TransactWriteItems` {% alert level="info" %} The `PutItem` operation requires additional configuration. For more information, see [Instrumenting Python Serverless Applications](https://docs.datadoghq.com/serverless/aws_lambda/installation/python/#span-auto-linking) or [Instrumenting Node.js Serverless Applications](https://docs.datadoghq.com/serverless/aws_lambda/installation/nodejs/#span-auto-linking). {% /alert %} ### S3 Change Notification Auto-linking{% #s3-change-notification-auto-linking %} For [S3 Change Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), Span Auto-linking supports the following operations: - `PutObject` - `CompleteMultipartUpload` - `CopyObject` ## Hybrid environments{% #hybrid-environments %} If you have installed Datadog's tracing libraries (`dd-trace`) on both your Lambda functions and hosts, your traces automatically show you the complete picture of requests that cross infrastructure boundaries, whether it be AWS Lambda, containers, on-prem hosts, or managed services. If `dd-trace` is installed on your hosts with the Datadog Agent, and your serverless functions are traced with AWS X-Ray, trace merging is required to see a single, connected trace across your infrastructure. See the [Serverless Trace Merging](https://docs.datadoghq.com/serverless/distributed_tracing/#trace-merging) documentation to learn more about merging traces from `dd-trace` and AWS X-Ray. Datadog's [AWS X-Ray integration](https://docs.datadoghq.com/integrations/amazon_xray/#overview) only provides traces for Lambda functions. See the [Datadog APM documentation](https://docs.datadoghq.com/tracing/trace_collection/) to learn more about tracing in container or host-based environments. ## Profiling your Lambda Functions{% #profiling-your-lambda-functions %} Datadog's [Continuous Profiler](https://docs.datadoghq.com/profiler/) is available in Preview for Python in version 4.62.0 and layer version 62 and above. This optional feature is enabled by setting the `DD_PROFILING_ENABLED` environment variable to `true`. The Continuous Profiler works by spawning a thread that periodically wakes up and takes a snapshot of the CPU and heap of all running Python code. This can include the profiler itself. If you want the profiler to ignore itself, set `DD_PROFILING_IGNORE_PROFILER` to `true`. ## Trace Merging{% #trace-merging %} ### Use cases{% #use-cases %} Datadog recommends using only the Datadog APM trace library (`dd-trace`), but in some advanced situations users can combine Datadog tracing and AWS X-Ray using trace merging. Trace merging is available for Node.js and Python AWS Lambda functions. If you aren't sure which tracing library to use, read about [choosing your tracing library](https://docs.datadoghq.com/serverless/distributed_tracing/). There are two primary reasons for instrumenting both `dd-trace` and AWS X-Ray tracing libraries: - In an AWS serverless environment, you are already tracing your Lambda functions with `dd-trace`, you require AWS X-Ray active tracing for AWS managed services such as AppSync and Step Functions, and you want to visualize the `dd-trace` and AWS X-Ray spans in one single trace. - In a hybrid environment with both Lambda functions and hosts, `dd-trace` instruments your hosts, AWS X-Ray instruments your Lambda functions, and you want to visualize connected traces for transactions across Lambda functions and hosts. **Note:** This may result in higher usage bills. X-Ray spans continue to be available in your merged traces after 2-5 minutes. In many cases, Datadog recommends only using a single tracing library. Learn more about [choosing your tracing library](https://docs.datadoghq.com/serverless/distributed_tracing/). You can find setup instructions for each of the above use cases below: - Trace merging in a serverless-first environment - Trace merging across AWS Lambda and hosts ### Trace merging in an AWS serverless environment{% #trace-merging-in-an-aws-serverless-environment %} AWS X-Ray provides both a backend AWS service (AWS X-Ray active tracing) and a set of client libraries. [Enabling the backend AWS service alone in the Lambda console](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) gives you `Initialization` and `Invocation` spans for your AWS Lambda functions. You can also enable AWS X-Ray active tracing from the API Gateway and Step Function consoles. Both the AWS X-Ray SDK and Datadog APM client libraries (`dd-trace`) add metadata and spans for downstream calls by accessing the function directly. Assuming you are using `dd-trace` to trace at the handler level, your setup should be similar to the following: 1. You have enabled [AWS X-Ray active tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) on your Lambda functions from the AWS Lambda console and our [AWS X-Ray integration within Datadog](https://app.datadoghq.com/account/settings#integrations/amazon_xray). 1. You have instrumented your Lambda functions with Datadog APM (`dd-trace`) by following the [installation instructions for your Lambda runtime](https://docs.datadoghq.com/serverless/installation/). 1. Third-party libraries are automatically patched by `dd-trace`, so the AWS X-Ray client libraries do not need to be installed. 1. Set the `DD_MERGE_XRAY_TRACES` environment variable to `true` on your Lambda functions to merge the X-Ray and `dd-trace` traces (`DD_MERGE_DATADOG_XRAY_TRACES` in Ruby). ### Tracing across AWS Lambda and hosts{% #tracing-across-aws-lambda-and-hosts %} #### Context propagation with the Datadog tracing libraries{% #context-propagation-with-the-datadog-tracing-libraries %} If you have installed Datadog's tracing libraries (`dd-trace`) on both your Lambda functions and hosts, your traces will automatically show you the complete picture of requests that cross infrastructure boundaries, whether it be AWS Lambda, containers, on-prem hosts, or managed services. #### Context propagation with the X-Ray integration{% #context-propagation-with-the-x-ray-integration %} If `dd-trace` is installed on your hosts with the Datadog Agent, and your Node.js or Python serverless functions are traced with AWS X-Ray, your setup should be similar to the following: 1. You have installed the [AWS X-Ray integration](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) for tracing your Lambda functions, enabling both AWS X-Ray active tracing and installing the X-Ray client libraries. 1. You have installed the [Datadog Lambda Library for your Lambda runtime](https://docs.datadoghq.com/serverless/installation/), and the `DD_TRACE_ENABLED` environment variable is set to `true`. 1. [Datadog APM](https://docs.datadoghq.com/tracing/send_traces/) is configured on your hosts and container-based infrastructure. Then, for X-Ray and Datadog APM traces to appear in the same flame graph, all services must have the same `env` tag. **Note**: Distributed Tracing is supported for any runtime for your host or container-based applications. Your hosts and Lambda functions do not need to be in the same runtime. {% image source="https://docs.dd-static.net/images/integrations/amazon_lambda/lambda_host_trace.96625ed5ae29c19fc8ce32184dca32e3.png?auto=format" alt="trace of a request from a host to a Lambda function" /%} ## Trace Propagation{% #trace-propagation %} {% image source="https://docs.dd-static.net/images/serverless/lambda-non-http-trace.72962e88af90b138af8d05ffaa96532e.png?auto=format" alt="Serverless Distributed Non-HTTP Trace" /%} ### Required setup{% #required-setup %} Additional instrumentation is sometimes required to see a single, connected trace in Node and Python serverless applications asynchronously triggering Lambda functions. If you are just getting started with monitoring serverless applications in Datadog, [follow our main installation steps](https://docs.datadoghq.com/serverless/installation) and [read this page on choosing your tracing library](https://docs.datadoghq.com/serverless/distributed_tracing). Once you are sending traces from your Lambda functions to Datadog using the [Datadog Lambda Library](https://docs.datadoghq.com/serverless/datadog_lambda_library), you may want to follow these steps to connect traces between two Lambda functions in cases such as: - Triggering Lambda functions via Step Functions - Invoking Lambda functions via non-HTTP protocols such as MQTT Tracing many AWS Managed services (listed [here](https://docs.datadoghq.com/serverless/distributed_tracing#runtime-recommendations)) is supported out-of-the-box and does not require following the steps outlined on this page. To successfully connect trace context between resources sending traces, you need to: - Include Datadog trace context in outgoing events. The outgoing event can originate from a host or Lambda function with `dd-trace` installed. - Extract the trace context in the consumer Lambda function. ### Passing trace context{% #passing-trace-context %} The following code samples outline how to pass trace context in outgoing payloads to services which do not support HTTP headers, or managed services not supported [natively](https://docs.datadoghq.com/serverless/distributed_tracing#runtime-recommendations) by Datadog in Node and Python: {% tab title="Python" %} In Python, you can use the `get_dd_trace_context` helper function to pass tracing context to outgoing events in a Lambda functions: ```py import json import boto3 import os from datadog_lambda.tracing import get_dd_trace_context # Datadog tracing helper function def handler(event, context): my_custom_client.sendRequest( { 'myCustom': 'data', '_datadog': { 'DataType': 'String', 'StringValue': json.dumps(get_dd_trace_context()) # Includes trace context in outgoing payload. }, }, ) ``` {% /tab %} {% tab title="Node.js" %} In Node, you can use the `getTraceHeaders` helper function to pass tracing context to outgoing events in a Lambda function: ```js const { getTraceHeaders } = require("datadog-lambda-js"); // Datadog tracing helper function module.exports.handler = async event => { const _datadog = getTraceHeaders(); // Captures current Datadog trace context. var payload = JSON.stringify({ data: 'sns', _datadog }); await myCustomClient.sendRequest(payload) ``` {% /tab %} #### From hosts{% #from-hosts %} If you aren't passing trace context from your Lambda functions, you can use the following code template in place of the `getTraceHeaders` and `get_dd_trace_context` helper functions to get the current span context. Instructions on how to do this in every runtime are outlined [here](https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/). ```js const tracer = require("dd-trace"); exports.handler = async event => { const span = tracer.scope().active(); const _datadog = {} tracer.inject(span, 'text_map', _datadog) // ... ``` ### Extracting trace context{% #extracting-trace-context %} To extract the above trace context from the consumer Lambda function, you need to define an extractor function that captures trace context before the execution of your Lambda function handler. To do this, configure the `DD_TRACE_EXTRACTOR` environment variable to point to the location of your extractor function. The format for this is `.`. For example, `extractors.json` if the `json` extractor is in the `extractors.js` file. Datadog recommends you place your extractor methods all in one file, as extractors can be re-used across multiple Lambda functions. These extractors are completely customizable to fit any use case. **Notes**: - If you are using TypeScript or a bundler like webpack, you must `import` or `require` your Node.js module where the extractors are defined. This ensures the module gets compiled and bundled into your Lambda deployment package. - If your Node.js Lambda function runs on `arm64`, you must [define the extractor in your function code](https://docs.datadoghq.com/serverless/guide/handler_wrapper/) instead of using the `DD_TRACE_EXTRACTOR` environment variable. #### Sample extractors{% #sample-extractors %} The following code samples outline sample extractors you might use for propagating trace context across a third party system, or an API which does not support standard HTTP headers. {% tab title="Python" %} ```py def extractor(payload): trace_headers = json.loads(payload["_datadog"]); trace_id = trace_headers["x-datadog-trace-id"]; parent_id = trace_headers["x-datadog-parent-id"]; sampling_priority = trace_headers["x-datadog-sampling-priority"]; return trace_id, parent_id, sampling_priority ``` {% /tab %} {% tab title="Node.js" %} ```js exports.json = (payload) => { const traceData = payload._datadog const traceID = traceData["x-datadog-trace-id"]; const parentID = traceData["x-datadog-parent-id"]; const sampledHeader = traceData["x-datadog-sampling-priority"]; const sampleMode = parseInt(sampledHeader, 10); return { parentID, sampleMode, source: 'event', traceID, }; }; ``` {% /tab %} {% tab title="Go" %} ```go var exampleSQSExtractor = func(ctx context.Context, ev json.RawMessage) map[string]string { eh := events.SQSEvent{} headers := map[string]string{} if err := json.Unmarshal(ev, &eh); err != nil { return headers } // Using SQS as a trigger with a batchSize=1 so it's important we check // for this as a single SQS message will drive the execution of the handler. if len(eh.Records) != 1 { return headers } record := eh.Records[0] lowercaseHeaders := map[string]string{} for k, v := range record.MessageAttributes { if v.StringValue != nil { lowercaseHeaders[strings.ToLower(k)] = *v.StringValue } } return lowercaseHeaders } cfg := &ddlambda.Config{ TraceContextExtractor: exampleSQSExtractor, } ddlambda.WrapFunction(handler, cfg) ``` {% /tab %} ## Sending traces to Datadog with the X-Ray Integration{% #sending-traces-to-datadog-with-the-x-ray-integration %} If you are already tracing your serverless application with X-Ray and want to continue using X-Ray, you can [install the AWS X-Ray integration](https://docs.datadoghq.com/integrations/amazon_xray/#overview) to send traces from X-Ray to Datadog. ## Further Reading{% #further-reading %} - [Explore Datadog APM](https://docs.datadoghq.com/tracing/) - [Live Search](https://docs.datadoghq.com/tracing/trace_search_and_analytics/#live-search-for-15-minutes) - [Real-time distributed tracing for Go and Java Lambda Functions](https://www.datadoghq.com/blog/aws-lambda-tracing-go-java-functions/) - [Monitor your serverless stack in the Serverless view](https://www.datadoghq.com/blog/datadog-serverless-view/) - [Datadog Serverless Monitoring for AWS fully managed services](https://www.datadoghq.com/blog/monitor-aws-fully-managed-services-datadog-serverless-monitoring/) - [Real-time distributed tracing for .NET Lambda functions](https://www.datadoghq.com/blog/dotnet-lambda-functions-distributed-tracing/)