Distributed Tracing

Distributed Tracing

By connecting your serverless traces to metrics, Datadog provides a context-rich picture of your application’s performance, allowing you to better troubleshoot performance issues given the distributed nature of serverless applications.

The Datadog Python, Node.js, Ruby, Go and Java tracing libraries support distributed tracing for AWS Lambda. The easiest way to add tracing to your application is with the Datadog Lambda Library, which includes the Datadog tracing library as a dependency.

Choose your tracing library

New to serverless monitoring? Follow the installation steps here to get started.

To start using Datadog APM with your serverless application, you can choose between installing Datadog’s tracing libraries (dd-trace) or AWS X-Ray tracing libraries based on your Lambda runtime and individual requirements. To see all of your traces in Datadog in real-time in the Live Search view, you need to use Datadog’s tracing libraries.

Datadog APM with dd-traceDatadog APM with AWS X-Ray
Uses Datadog APM’s integration libraries for end-to-end tracing.Pulls traces from AWS X-Ray.
Visualize your traces in Datadog in real-time.Trace data available in Datadog after a few minutes.
Tail-based sampling and fully customizable tag-based retention filters.Sampling rate cannot be configured.
Support for Python, Node.js, Ruby, Go, Java.Support for all Lambda runtimes.

Runtime recommendations


Python and Node.js

The Datadog Lambda Library and tracing libraries for Python and Node.js support:

  • Automatic correlation of Lambda logs and traces with trace ID and tag injection.
  • Installation without any code changes using Serverless Framework, AWS SAM and AWS CDK integrations.
  • Tracing HTTP requests invoking downstream Lambda functions or containers.
  • Tracing consecutive Lambda invocations made via the AWS SDK.
  • Tracing asynchronous Lambda invocations through AWS SQS.
  • Tracing dozens of additional out-of-the-box Python and Node.js libraries.
  • Tracing non-HTTP requests made through SNS, Kinesis, EventBridge, MQTT and more (requires additional instrumention outlined here).

For Python and Node.js serverless applications, Datadog recommends you install Datadog’s tracing libraries. If your application requires AWS X-Ray active tracing in AWS managed services such as API Gateway or Step Functions, we recommend you augment AWS X-Ray traces with Datadog APM by configuring both AWS X-Ray and Datadog APM tracing libraries as described here.

If you are already tracing your serverless functions with X-Ray and want to continue using X-Ray, you can install our AWS X-Ray integration.

Looking to trace through serverless resources not listed above? Open a feature request here.

Ruby

The Datadog Lambda Library and tracing libraries for Ruby support:

  • Automatic correlation of Lambda logs and traces with trace ID and tag injection.
  • Tracing HTTP requests invoking downstream Lambda functions or containers.
  • Tracing dozens of additional out-of-the-box Ruby libraries.

You can trace your serverless functions in Datadog with Datadog’s tracing libraries or by installing our AWS X-Ray integration. If you are using Datadog’s tracing libraries, and need to connect Lambda function traces across AWS managed services, we recommend you augment your traces by configuring both AWS X-Ray and Datadog APM tracing libraries as described here.

Looking to trace through serverless resources not listed above? Open a feature request here.

Go

The Datadog Lambda Library and tracing libraries for Go support:

  • Manual correlation of Lambda logs and traces with trace ID and tag injection.
  • Tracing HTTP requests invoking downstream Lambda functions or containers.
  • Tracing dozens of additional out-of-the-box Go libraries.

You can trace your serverless functions in Datadog with Datadog’s tracing libraries or by installing our AWS X-Ray integration. If you are using Datadog’s tracing libraries, and need to connect multiple Lambda function traces in event-driven architectures, we recommend you augment your traces by configuring both AWS X-Ray and Datadog APM tracing libraries as described here.

Looking to trace through serverless resources not listed above? Open a feature request here.

Java

The Datadog Lambda Library and tracing libraries for Java support:

  • Correlation of Lambda logs and traces with trace ID and tag injection. See Connecting Java logs and traces for more details.
  • Tracing HTTP requests invoking downstream Lambda functions or containers.
  • Tracing dozens of additional out-of-the-box Java libraries.

You can trace your serverless functions in Datadog with Datadog’s tracing libraries or by installing the AWS X-Ray integration. If you are using Datadog’s tracing libraries, and you need to connect multiple Lambda function traces in event-driven architectures, it is recommended that you augment your traces by configuring both AWS X-Ray and Datadog APM tracing libraries, as described in the serverless trace merging documentation.

Have feedback on the Datadog’s tracing libraries for Java Lambda functions? Make sure to check out discussions going on in the #serverless channel in the Datadog Slack community.

.NET

Datadog recommends you configure tracing for your .NET AWS Lambda serverless applications by installing our AWS X-Ray integration.

Learn more about tracing through .NET Azure serverless applications here.

Looking to trace through .NET AWS Lambda serverless applications with Datadog tracing libraries? Open a feature request here.

Hybrid environments

If you have installed Datadog’s tracing libraries (dd-trace) on both your Lambda functions and hosts, your traces will automatically show you the complete picture of requests that cross infrastructure boundaries, whether it be AWS Lambda, containers, on-prem hosts, or managed services.

If dd-trace is installed on your hosts with the Datadog Agent, and your serverless functions are traced with AWS X-Ray, trace merging is required to see a single, connected trace across your infrastructure. Learn more about merging traces from dd-trace and AWS X-Ray here.

Datadog’s AWS X-Ray integration only provides traces for Lambda functions. Learn more about tracing in container or host-based environments here.

Enable Datadog APM

Live Search

The Datadog Python, Node.js, Ruby, Go and Java tracing libraries support distributed tracing for AWS Lambda. To enable tracing on your functions, follow the installation instructions.

To enable Datadog APM without enabling logging for your functions, ensure the DdForwarderLog environment variable is set to false on your Datadog Forwarder.

Further Reading

Additional helpful documentation, links, and articles: