--- title: Distributed Tracing with AWS Lambda Serverless Applications description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Serverless > Serverless Monitoring for AWS Lambda > Distributed Tracing with AWS Lambda Serverless Applications --- # Distributed Tracing with AWS Lambda Serverless Applications {% image source="https://docs.dd-static.net/images/tracing/serverless_functions/ServerlessDistributedTrace.8849d3898d8b3c33b995508247c9dc7c.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/tracing/serverless_functions/ServerlessDistributedTrace.8849d3898d8b3c33b995508247c9dc7c.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Trace Serverless Functions" /%} By connecting your serverless traces to metrics, Datadog provides a context-rich picture of your application's performance, allowing you to better troubleshoot performance issues given the distributed nature of serverless applications. The Datadog Python, Node.js, Ruby, Go, Java, and .NET SDKs support distributed tracing for AWS Lambda. ## Send traces from your serverless application{% #send-traces-from-your-serverless-application %} {% image source="https://docs.dd-static.net/images/serverless/serverless_tracing_installation_instructions.916e542d516b6e0e19d5f1dcf45dc027.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/serverless/serverless_tracing_installation_instructions.916e542d516b6e0e19d5f1dcf45dc027.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Architecture diagram for tracing AWS Lambda with Datadog" /%} The Datadog Python, Node.js, Ruby, Go, Java, and .NET SDKs support distributed tracing for AWS Lambda. You can install the SDK using the [installation instructions](https://docs.datadoghq.com/serverless/installation.md). ### Runtime recommendations{% #runtime-recommendations %} - [Python](https://docs.datadoghq.com/serverless/distributed_tracing#python-and-nodejs) - [Node.js](https://docs.datadoghq.com/serverless/distributed_tracing#python-and-nodejs) - [Ruby](https://docs.datadoghq.com/serverless/distributed_tracing#ruby) - [Java](https://docs.datadoghq.com/serverless/distributed_tracing#java) - [go](https://docs.datadoghq.com/serverless/distributed_tracing#go) - [.NET](https://docs.datadoghq.com/serverless/distributed_tracing#net) #### Python and Node.js{% #python-and-nodejs %} The Datadog Lambda Library and SDKs for Python and Node.js support: - Automatic correlation of Lambda logs and traces with trace ID and tag injection. - Installation without any code changes using Serverless Framework, AWS SAM and AWS CDK integrations. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing consecutive Lambda invocations made via the AWS SDK. - Cold start tracing - Tracing asynchronous Lambda invocations through AWS Managed Services - API Gateway - SQS - SNS - SNS and SQS direct integration - Kinesis - EventBridge - DynamoDB - S3 - Step Functions - Tracing dozens of additional out-of-the-box [Python](https://docs.datadoghq.com/tracing/trace_collection/compatibility/python.md) and [Node.js](https://docs.datadoghq.com/tracing/trace_collection/compatibility/nodejs.md) libraries. For Python and Node.js serverless applications, Datadog recommends you [install Datadog SDKs](https://docs.datadoghq.com/serverless/installation.md). *Looking to trace through serverless resources not listed above? [Open a feature request](https://docs.datadoghq.com/help/).* #### Ruby{% #ruby %} The Datadog Lambda Library and SDKs for Ruby support: - Automatic correlation of Lambda logs and traces with trace ID and tag injection. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [Ruby](https://docs.datadoghq.com/tracing/trace_collection/compatibility/ruby.md) libraries. You can trace your serverless functions in Datadog with [Datadog SDKs](https://docs.datadoghq.com/serverless/installation.md). *Looking to trace through serverless resources not listed above? [Open a feature request](https://docs.datadoghq.com/help/).* #### Go{% #go %} The Datadog Lambda Library and SDKs for Go support: - Manual correlation of Lambda logs and traces with trace ID and tag injection. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [Go](https://docs.datadoghq.com/tracing/trace_collection/compatibility/go.md) libraries. For Go serverless applications, Datadog recommends installing [Datadog SDKs](https://docs.datadoghq.com/serverless/installation.md). *Looking to trace through serverless resources not listed above? [Open a feature request](https://docs.datadoghq.com/help/).* #### Java{% #java %} The Datadog Lambda Library and SDKs for Java support: - Correlation of Lambda logs and traces with trace ID and tag injection. See [Connecting Java logs and traces](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/java.md) for more details. - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [Java](https://docs.datadoghq.com/tracing/trace_collection/compatibility/java.md) libraries. For Java serverless applications, Datadog recommends [installing Datadog SDKs](https://docs.datadoghq.com/serverless/installation.md). *Have feedback on the Datadog SDKs for Java Lambda functions? Make sure to check out discussions going on in the [\#serverless](https://datadoghq.slack.com/archives/CFDPB83M4) channel in the [Datadog Slack community](https://chat.datadoghq.com/).* #### .NET{% #net %} The SDK for .NET supports: - Tracing HTTP requests invoking downstream Lambda functions or containers. - Tracing dozens of additional out-of-the-box [.NET](https://docs.datadoghq.com/tracing/trace_collection/compatibility/dotnet-core.md) libraries. For .NET serverless applications, Datadog recommends [installing Datadog SDKs](https://docs.datadoghq.com/serverless/installation.md). Learn more about [tracing through .NET Azure serverless applications](https://docs.datadoghq.com/serverless/azure_app_services.md). ## Span Auto-linking{% #span-auto-linking %} {% image source="https://docs.dd-static.net/images/serverless/lambda/tracing/autolink.fa059aab20b6ec455211424c2d24a0ef.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/serverless/lambda/tracing/autolink.fa059aab20b6ec455211424c2d24a0ef.png?auto=format&fit=max&w=850&dpr=2 2x" alt="In Datadog, a DynamoDB trace. At the top, a message reads 'This trace is linked to other traces'. The Span Links tab is open and displays a clickable link to another DynamoDB trace." /%} Datadog automatically detects linked spans when segments of your asynchronous requests cannot propagate trace context. For example, this may occur when a request triggers an [S3 Change Events](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), or [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html). You can see Auto-linked spans appear in the [Span Links tab](https://docs.datadoghq.com/tracing/trace_explorer/trace_view.md?tab=spanlinksbeta). These appear as **Backward** or **Forward**. *Backward*: The linked span was caused by the trace you are viewing. *Forward*: The linked span caused the trace you are viewing. {% alert level="info" %} Sampling and [trace retention filters](https://docs.datadoghq.com/tracing/trace_pipeline/trace_retention.md) can interfere with Auto-linking. To improve your chances of seeing Auto-linked spans, increase your sample rate or adjust your retention filters. {% /alert %} ### Supported technologies{% #supported-technologies %} Span Auto-linking is available for: - Python AWS Lambda functions instrumented with [`datadog-lambda-python`](https://github.com/DataDog/datadog-lambda-python) layer v101+ - Python applications instrumented with [`dd-trace-py`](https://github.com/DataDog/dd-trace-py/) v2.16+ - Node.js AWS Lambda functions instrumented with [`datadog-lambda-js`](https://github.com/DataDog/datadog-lambda-js) layer 118+ - Node.js applications instrumented with [`dd-trace-js`](https://github.com/DataDog/dd-trace-js/) v4.53.0+ or v5.29.0+ ### DynamoDB Change Stream Auto-linking{% #dynamodb-change-stream-auto-linking %} For [DynamoDB Change Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html), Span Auto-linking supports the following operations: - `PutItem` - `UpdateItem` - `DeleteItem` - `BatchWriteItem` - `TransactWriteItems` {% alert level="info" %} The `PutItem` operation requires additional configuration. For more information, see [Instrumenting Python Serverless Applications](https://docs.datadoghq.com/serverless/aws_lambda/installation/python.md#span-auto-linking) or [Instrumenting Node.js Serverless Applications](https://docs.datadoghq.com/serverless/aws_lambda/installation/nodejs.md#span-auto-linking). {% /alert %} ### S3 Change Notification Auto-linking{% #s3-change-notification-auto-linking %} For [S3 Change Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html), Span Auto-linking supports the following operations: - `PutObject` - `CompleteMultipartUpload` - `CopyObject` ## Hybrid environments{% #hybrid-environments %} For end-to-end visibility across Lambda functions, hosts, containers, and managed services, install the Datadog SDKs (`dd-trace`) on both your Lambda functions and your hosts. Your traces then show a complete picture of requests that cross infrastructure boundaries. On Lambda, install `dd-trace` with the [Datadog Lambda Extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension.md), which runs the Datadog Agent inside the Lambda execution environment and ships traces directly to Datadog with minimal overhead. The Lambda Extension is the recommended installation method for new and existing serverless applications. See the [Datadog APM documentation](https://docs.datadoghq.com/tracing/trace_collection.md) for tracing setup in container and host-based environments. ## Profiling your Lambda Functions{% #profiling-your-lambda-functions %} Datadog's [Continuous Profiler](https://docs.datadoghq.com/profiler.md) is available in Preview for Python in version 4.62.0 and layer version 62 and above. This optional feature is enabled by setting the `DD_PROFILING_ENABLED` environment variable to `true`. The Continuous Profiler works by spawning a thread that periodically wakes up and takes a snapshot of the CPU and heap of all running Python code. This can include the profiler itself. If you want the profiler to ignore itself, set `DD_PROFILING_IGNORE_PROFILER` to `true`. ## Trace Merging{% #trace-merging %} ### Use cases{% #use-cases %} Datadog recommends using only the Datadog APM trace library (`dd-trace`), but in some advanced situations users can combine Datadog tracing and AWS X-Ray using trace merging. Trace merging is available for Node.js and Python AWS Lambda functions. If you aren't sure which SDK to use, read about [choosing your SDK](https://docs.datadoghq.com/serverless/distributed_tracing.md). {% alert level="info" %} AWS Step Functions tracing is supported natively by Datadog and no longer requires X-Ray. See [Serverless Monitoring for AWS Step Functions](https://docs.datadoghq.com/serverless/step_functions.md) and [Merge Step Functions and Lambda Traces](https://docs.datadoghq.com/serverless/step_functions/merge-step-functions-lambda.md). {% /alert %} There are two primary reasons for instrumenting both `dd-trace` and AWS X-Ray tracing libraries: - In an AWS serverless environment, you are already tracing your Lambda functions with `dd-trace`, you require AWS X-Ray active tracing for an AWS managed service that Datadog APM doesn't yet instrument (such as AppSync), and you want to visualize the `dd-trace` and AWS X-Ray spans in one single trace. - In a hybrid environment with both Lambda functions and hosts, `dd-trace` instruments your hosts, AWS X-Ray instruments your Lambda functions, and you want to visualize connected traces for transactions across Lambda functions and hosts. **Note:** This may result in higher usage bills. X-Ray spans continue to be available in your merged traces after 2-5 minutes. In many cases, Datadog recommends only using a single SDK. Learn more about [choosing your SDK](https://docs.datadoghq.com/serverless/distributed_tracing.md). You can find setup instructions for each of the above use cases below: - Trace merging in a serverless-first environment - Trace merging across AWS Lambda and hosts ### Trace merging in an AWS serverless environment{% #trace-merging-in-an-aws-serverless-environment %} AWS X-Ray provides both a backend AWS service (AWS X-Ray active tracing) and a set of client libraries. [Enabling the backend AWS service alone in the Lambda console](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) gives you `Initialization` and `Invocation` spans for your AWS Lambda functions. You can also enable AWS X-Ray active tracing from the API Gateway and Step Function consoles. Both the AWS X-Ray SDK and Datadog APM client libraries (`dd-trace`) add metadata and spans for downstream calls by accessing the function directly. Assuming you are using `dd-trace` to trace at the handler level, your setup should be similar to the following: 1. You have enabled [AWS X-Ray active tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) on your Lambda functions from the AWS Lambda console and our [AWS X-Ray integration within Datadog](https://app.datadoghq.com/account/settings#integrations/amazon_xray). 1. You have instrumented your Lambda functions with Datadog APM (`dd-trace`) by following the [installation instructions for your Lambda runtime](https://docs.datadoghq.com/serverless/installation.md). 1. Third-party libraries are automatically patched by `dd-trace`, so the AWS X-Ray client libraries do not need to be installed. 1. Set the `DD_MERGE_XRAY_TRACES` environment variable to `true` on your Lambda functions to merge the X-Ray and `dd-trace` traces (`DD_MERGE_DATADOG_XRAY_TRACES` in Ruby). ### Tracing across AWS Lambda and hosts{% #tracing-across-aws-lambda-and-hosts %} #### Context propagation with the Datadog SDKs (recommended){% #context-propagation-with-the-datadog-sdks-recommended %} Install Datadog SDKs (`dd-trace`) on both your Lambda functions and hosts. Your traces then automatically show a complete picture of requests that cross infrastructure boundaries, whether it be AWS Lambda, containers, on-prem hosts, or managed services. {% image source="https://docs.dd-static.net/images/integrations/amazon_lambda/lambda_host_trace.96625ed5ae29c19fc8ce32184dca32e3.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/integrations/amazon_lambda/lambda_host_trace.96625ed5ae29c19fc8ce32184dca32e3.png?auto=format&fit=max&w=850&dpr=2 2x" alt="trace of a request from a host to a Lambda function" /%} ## Trace Propagation{% #trace-propagation %} {% image source="https://docs.dd-static.net/images/serverless/lambda-non-http-trace.72962e88af90b138af8d05ffaa96532e.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/serverless/lambda-non-http-trace.72962e88af90b138af8d05ffaa96532e.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Serverless Distributed Non-HTTP Trace" /%} ### Required setup{% #required-setup %} Additional instrumentation is sometimes required to see a single, connected trace in Node and Python serverless applications asynchronously triggering Lambda functions. If you are just getting started with monitoring serverless applications in Datadog, [follow our main installation steps](https://docs.datadoghq.com/serverless/installation.md) and [read this page on choosing your SDK](https://docs.datadoghq.com/serverless/distributed_tracing.md). Once you are sending traces from your Lambda functions to Datadog using the [Datadog Lambda Library](https://docs.datadoghq.com/serverless/datadog_lambda_library.md), you may want to follow these steps to connect traces between two Lambda functions in cases such as: - Triggering Lambda functions via Step Functions - Invoking Lambda functions via non-HTTP protocols such as MQTT Tracing many AWS Managed services (listed [here](https://docs.datadoghq.com/serverless/distributed_tracing.md#runtime-recommendations)) is supported out-of-the-box and does not require following the steps outlined on this page. To successfully connect trace context between resources sending traces, you need to: - Include Datadog trace context in outgoing events. The outgoing event can originate from a host or Lambda function with `dd-trace` installed. - Extract the trace context in the consumer Lambda function. ### Passing trace context{% #passing-trace-context %} The following code samples outline how to pass trace context in outgoing payloads to services which do not support HTTP headers, or managed services not supported [natively](https://docs.datadoghq.com/serverless/distributed_tracing.md#runtime-recommendations) by Datadog in Node and Python: {% tab title="Python" %} In Python, you can use the `get_dd_trace_context` helper function to pass tracing context to outgoing events in a Lambda functions: ```py import json import boto3 import os from datadog_lambda.tracing import get_dd_trace_context # Datadog tracing helper function def handler(event, context): my_custom_client.sendRequest( { 'myCustom': 'data', '_datadog': { 'DataType': 'String', 'StringValue': json.dumps(get_dd_trace_context()) # Includes trace context in outgoing payload. }, }, ) ``` {% /tab %} {% tab title="Node.js" %} In Node, you can use the `getTraceHeaders` helper function to pass tracing context to outgoing events in a Lambda function: ```js const { getTraceHeaders } = require("datadog-lambda-js"); // Datadog tracing helper function module.exports.handler = async event => { const _datadog = getTraceHeaders(); // Captures current Datadog trace context. var payload = JSON.stringify({ data: 'sns', _datadog }); await myCustomClient.sendRequest(payload) ``` {% /tab %} #### From hosts{% #from-hosts %} If you aren't passing trace context from your Lambda functions, you can use the following code template in place of the `getTraceHeaders` and `get_dd_trace_context` helper functions to get the current span context. Instructions on how to do this in every runtime are outlined [here](https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation.md). ```js const tracer = require("dd-trace"); exports.handler = async event => { const span = tracer.scope().active(); const _datadog = {} tracer.inject(span, 'text_map', _datadog) // ... ``` ### Extracting trace context{% #extracting-trace-context %} To extract the above trace context from the consumer Lambda function, you need to define an extractor function that captures trace context before the execution of your Lambda function handler. To do this, configure the `DD_TRACE_EXTRACTOR` environment variable to point to the location of your extractor function. The format for this is `.`. For example, `extractors.json` if the `json` extractor is in the `extractors.js` file. Datadog recommends you place your extractor methods all in one file, as extractors can be re-used across multiple Lambda functions. These extractors are completely customizable to fit any use case. **Notes**: - If you are using TypeScript or a bundler like webpack, you must `import` or `require` your Node.js module where the extractors are defined. This ensures the module gets compiled and bundled into your Lambda deployment package. - If your Node.js Lambda function runs on `arm64`, you must [define the extractor in your function code](https://docs.datadoghq.com/serverless/guide/handler_wrapper.md) instead of using the `DD_TRACE_EXTRACTOR` environment variable. #### Sample extractors{% #sample-extractors %} The following code samples outline sample extractors you might use for propagating trace context across a third party system, or an API which does not support standard HTTP headers. {% tab title="Python" %} ```py def extractor(payload): trace_headers = json.loads(payload["_datadog"]); trace_id = trace_headers["x-datadog-trace-id"]; parent_id = trace_headers["x-datadog-parent-id"]; sampling_priority = trace_headers["x-datadog-sampling-priority"]; return trace_id, parent_id, sampling_priority ``` {% /tab %} {% tab title="Node.js" %} ```js exports.json = (payload) => { const traceData = payload._datadog const traceID = traceData["x-datadog-trace-id"]; const parentID = traceData["x-datadog-parent-id"]; const sampledHeader = traceData["x-datadog-sampling-priority"]; const sampleMode = parseInt(sampledHeader, 10); return { parentID, sampleMode, source: 'event', traceID, }; }; ``` {% /tab %} {% tab title="Go" %} ```go var exampleSQSExtractor = func(ctx context.Context, ev json.RawMessage) map[string]string { eh := events.SQSEvent{} headers := map[string]string{} if err := json.Unmarshal(ev, &eh); err != nil { return headers } // Using SQS as a trigger with a batchSize=1 so it's important we check // for this as a single SQS message will drive the execution of the handler. if len(eh.Records) != 1 { return headers } record := eh.Records[0] lowercaseHeaders := map[string]string{} for k, v := range record.MessageAttributes { if v.StringValue != nil { lowercaseHeaders[strings.ToLower(k)] = *v.StringValue } } return lowercaseHeaders } cfg := &ddlambda.Config{ TraceContextExtractor: exampleSQSExtractor, } ddlambda.WrapFunction(handler, cfg) ``` {% /tab %} ## Sending traces to Datadog with the X-Ray Integration{% #sending-traces-to-datadog-with-the-x-ray-integration %} If you have existing X-Ray instrumentation and want to keep using it, [install the AWS X-Ray integration](https://docs.datadoghq.com/integrations/amazon_xray.md#overview) to send traces from X-Ray to Datadog. For new serverless applications, Datadog recommends instrumenting Lambda functions with the [Datadog Lambda Extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension.md) instead. ## Further Reading{% #further-reading %} - [Explore Datadog APM](https://docs.datadoghq.com/tracing.md) - [Live Search](https://docs.datadoghq.com/tracing/trace_search_and_analytics.md#live-search-for-15-minutes) - [Real-time distributed tracing for Go and Java Lambda Functions](https://www.datadoghq.com/blog/aws-lambda-tracing-go-java-functions/) - [Monitor your serverless stack in the Serverless view](https://www.datadoghq.com/blog/datadog-serverless-view/) - [Datadog Serverless Monitoring for AWS fully managed services](https://www.datadoghq.com/blog/monitor-aws-fully-managed-services-datadog-serverless-monitoring/) - [Real-time distributed tracing for .NET Lambda functions](https://www.datadoghq.com/blog/dotnet-lambda-functions-distributed-tracing/)