Trace Context Propagation
Trace Context propagation is the mechanism of passing tracing information like Trace ID, Span ID, and sampling decisions from one part of a distributed application to another. This enables all traces (and additional telemetry) in a request to be correlated. When automatic instrumentation is enabled, trace context propagation is handled automatically by the APM SDK.
By default, the Datadog SDK extracts and injects distributed tracing headers using the following formats:
This default configuration maximizes compatibility with older Datadog SDK versions and products while allowing interoperability with other distributed tracing systems like OpenTelemetry.
Customize trace context propagation
You may need to customize the trace context propagation configuration if your applications:
- Communicate distributed tracing information in a different supported format
- Need to prevent extracting or injecting distributed tracing headers
Use the following environment variables to configure formats for reading and writing distributed tracing headers. Refer to the Language support section for language-specific configuration values.
DD_TRACE_PROPAGATION_STYLE
- Specifies trace context propagation formats for extraction and injection in a comma-separated list. May be overridden by extract-specific or inject-specific configurations.
Default: datadog,tracecontext
Note: With multiple formats, extraction follows the specified order (for example, datadog,tracecontext
checks Datadog headers first). The first valid context continues the trace; additional valid contexts become span links. OTEL_PROPAGATORS
- Specifies trace context propagation formats for both extraction and injection (comma-separated list). Lowest precedence; ignored if any other Datadog trace context propagation environment variable is set.
Note: Only use this configuration when migrating an application from the OpenTelemetry SDK to the Datadog SDK. For more information on this configuration and other OpenTelemetry environment variables, see Using OpenTelemetry Environment Variables with Datadog SDKs.
Advanced configuration
Most services send and receive trace context headers using the same format. However, if your service needs to accept trace context headers in one format and send them in another, use these configurations:
DD_TRACE_PROPAGATION_STYLE_EXTRACT
- Specifies trace context propagation formats for extraction only in a comma-separated list. Highest precedence for configuring extraction propagators.
DD_TRACE_PROPAGATION_STYLE_INJECT
- Specifies trace context propagation formats for injection only in comma-separated list. Highest precedence for configuring injection propagators.
The Datadog SDK supports the following trace context formats:
Language support
The Datadog Java SDK supports the following trace context formats, including deprecated configuration values:
Additional configuration
In addition to the environment variable configuration, you can also update the propagators using System Property configuration:
-Ddd.trace.propagation.style=datadog,b3multi
-Dotel.propagators=datadog,b3multi
-Ddd.trace.propagation.style.inject=datadog,b3multi
-Ddd.trace.propagation.style.extract=datadog,b3multi
The Datadog Python SDK supports the following trace context formats, including deprecated configuration values:
The Datadog Ruby SDK supports the following trace context formats, including deprecated configuration values:
Additional configuration
In addition to the environment variable configuration, you can also update the propagators in code by using Datadog.configure
:
Datadog.configure do |c|
# List of header formats that should be extracted
c.tracing.propagation_extract_style = [ 'tracecontext', 'datadog', 'b3' ]
# List of header formats that should be injected
c.tracing.propagation_inject_style = [ 'tracecontext', 'datadog' ]
end
The Datadog Go SDK supports the following trace context formats, including deprecated configuration values:
The Datadog Node.js SDK supports the following trace context formats, including deprecated configuration values:
The Datadog PHP SDK supports the following trace context formats, including deprecated configuration values:
Additional use cases
The following use cases are specific to the Datadog PHP SDK:
When a new PHP script is launched, the Datadog SDK automatically checks for the presence of Datadog headers for distributed tracing:
x-datadog-trace-id
(environment variable: HTTP_X_DATADOG_TRACE_ID
)x-datadog-parent-id
(environment variable: HTTP_X_DATADOG_PARENT_ID
)x-datadog-origin
(environment variable: HTTP_X_DATADOG_ORIGIN
)x-datadog-tags
(environment variable: HTTP_X_DATADOG_TAGS
)
To manually set tracing information in a CLI script for new or existing traces, use the DDTrace\set_distributed_tracing_context(string $trace_id, string $parent_id, ?string $origin = null, ?array $tags = null)
function.
<?php
function processIncomingQueueMessage($message) {
}
\DDTrace\trace_function(
'processIncomingQueueMessage',
function(\DDTrace\SpanData $span, $args) {
$message = $args[0];
\DDTrace\set_distributed_tracing_context($message->trace_id, $message->parent_id);
}
);
For version 0.87.0 and later, if the raw headers are available, use the DDTrace\consume_distributed_tracing_headers(array|callable $headersOrCallback)
function. Note: The header names must be in lowercase.
$headers = [
"x-datadog-trace-id" => "1234567890",
"x-datadog-parent-id" => "987654321",
];
\DDTrace\consume_distributed_tracing_headers($headers);
To extract the trace context directly as headers, use the DDTrace\generate_distributed_tracing_headers(?array $inject = null): array
function.
$headers = DDTrace\generate_distributed_tracing_headers();
// Store headers somewhere, inject them in an outbound request, ...
// These $headers can also be read back by \DDTrace\consume_distributed_tracing_headers from another process.
This function’s optional argument accepts an array of injection style names. It defaults to the configured injection style.
The PHP APM SDK supports automatic tracing of the php-amqplib/php-amqplib
library (version 0.87.0+). However, in some cases, your distributed trace may be disconnected. For example, when reading messages from a distributed queue using the basic_get
method outside an existing trace, you need to add a custom trace around the basic_get
call and corresponding message processing:
// Create a surrounding trace
$newTrace = \DDTrace\start_trace_span();
$newTrace->name = 'basic_get.process';
$newTrace->service = 'amqp';
// basic_get call(s) + message(s) processing
$msg = $channel->basic_get($queue);
if ($msg) {
$messageProcessing($msg);
}
// Once done, close the span
\DDTrace\close_span();
Creating this surrounding trace to your consuming-processing logic ensures observability of your distributed queue.
The Datadog C++ SDK supports the following trace context formats, including deprecated configuration values:
Additional configuration
In addition to the environment variable configuration, you can also update the propagators in code:
#include <datadog/tracer_config.h>
#include <datadog/propagation_style.h>
namespace dd = datadog::tracing;
int main() {
dd::TracerConfig config;
config.service = "my-service";
// `injection_styles` indicates with which tracing systems trace propagation
// will be compatible when injecting (sending) trace context.
// All styles indicated by `injection_styles` are used for injection.
// `injection_styles` is overridden by the `DD_TRACE_PROPAGATION_STYLE_INJECT`
// and `DD_TRACE_PROPAGATION_STYLE` environment variables.
config.injection_styles = {dd::PropagationStyle::DATADOG, dd::PropagationStyle::B3};
// `extraction_styles` indicates with which tracing systems trace propagation
// will be compatible when extracting (receiving) trace context.
// Extraction styles are applied in the order in which they appear in
// `extraction_styles`. The first style that produces trace context or
// produces an error determines the result of extraction.
// `extraction_styles` is overridden by the
// `DD_TRACE_PROPAGATION_STYLE_EXTRACT` and `DD_TRACE_PROPAGATION_STYLE`
// environment variables.
config.extraction_styles = {dd::PropagationStyle::W3C};
...
}
Additional use cases
To extract propagation context, implement a custom DictReader
interface and call Tracer::extract_span
or Tracer::extract_or_create_span
.
Here is an example of extracting propagation context from HTTP Headers:
#include <datadog/dict_reader.h>
#include <datadog/optional.h>
#include <datadog/string_view.h>
#include <unordered_map>
namespace dd = datadog::tracing;
class HTTPHeadersReader : public datadog::tracing::DictReader {
std::unordered_map<dd::StringView, dd::StringView> headers_;
public:
HTTPHeadersReader(std::unordered_map<dd::StringView, dd::StringView> headers)
: headers_(std::move(headers)) {}
~HTTPHeadersReader() override = default;
// Return the value at the specified `key`, or return `nullopt` if there
// is no value at `key`.
dd::Optional<dd::StringView> lookup(dd::StringView key) const override {
auto found = headers_.find(key);
if (found == headers_.cend()) return dd::nullopt;
return found->second;
}
// Invoke the specified `visitor` once for each key/value pair in this object.
void visit(
const std::function<void(dd::StringView key, dd::StringView value)>& visitor)
const override {
for (const auto& [key, value] : headers_) {
visitor(key, value);
}
};
};
// Usage example:
void handle_http_request(const Request& request, datadog::tracing::Tracer& tracer) {
HTTPHeadersReader reader{request.headers};
auto maybe_span = tracer.extract_span(reader);
..
}
To inject propagation context, implement the DictWriter
interface and call Span::inject
on a span instance:
#include <datadog/dict_writer.h>
#include <datadog/string_view.h>
#include <string>
#include <unordered_map>
using namespace dd = datadog::tracing;
class HTTPHeaderWriter : public dd::DictWriter {
std::unordered_map<std::string, std::string>& headers_;
public:
explicit HTTPHeaderWriter(std::unordered_map<std::string, std::string>& headers) : headers_(headers) {}
~HTTPHeaderWriter() override = default;
void set(dd::StringView key, dd::StringView value) override {
headers_.emplace(key, value);
}
};
// Usage example:
void handle_http_request(const Request& request, dd::Tracer& tracer) {
auto span = tracer.create_span();
HTTPHeaderWriter writer(request.headers);
span.inject(writer);
// `request.headers` now populated with the headers needed to propagate the span.
..
}
The Datadog .NET SDK supports the following trace context formats, including deprecated configuration values:
Additional use cases
- As of version [2.48.0][6], the default propagation style is
datadog, tracecontext
. This means Datadog headers are used first, followed by W3C Trace Context. - Prior to version 2.48.0, the order was
tracecontext, Datadog
for both extraction and injection propagation. - Prior to version [2.22.0][7], only the
Datadog
injection style was enabled. - As of version [2.42.0][8], when multiple extractors are specified, the
DD_TRACE_PROPAGATION_EXTRACT_FIRST=true
configuration specifies whether context extraction should exit immediately upon detecting the first valid tracecontext
. The default value is false
.
In most cases, header extraction and injection are automatic. However, there are some known cases where your distributed trace can be disconnected. For instance, when reading messages from a distributed queue, some libraries may lose the span context. It also happens if you set DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED
to false
when consuming Kafka messages. In these cases, you can add a custom trace using the following code:
var spanContextExtractor = new SpanContextExtractor();
var parentContext = spanContextExtractor.Extract(headers, (headers, key) => GetHeaderValues(headers, key));
var spanCreationSettings = new SpanCreationSettings() { Parent = parentContext };
using var scope = Tracer.Instance.StartActive("operation", spanCreationSettings);
Provide the GetHeaderValues
method. The way this method is implemented depends on the structure that carries SpanContext
.
Here are some examples:
// Confluent.Kafka
IEnumerable<string> GetHeaderValues(Headers headers, string name)
{
if (headers.TryGetLastBytes(name, out var bytes))
{
try
{
return new[] { Encoding.UTF8.GetString(bytes) };
}
catch (Exception)
{
// ignored
}
}
return Enumerable.Empty<string>();
}
// RabbitMQ
IEnumerable<string> GetHeaderValues(IDictionary<string, object> headers, string name)
{
if (headers.TryGetValue(name, out object value) && value is byte[] bytes)
{
return new[] { Encoding.UTF8.GetString(bytes) };
}
return Enumerable.Empty<string>();
}
// SQS
public static IEnumerable<string> GetHeaderValues(IDictionary<string, MessageAttributeValue> headers, string name)
{
// For SQS, there are a maximum of 10 message attribute headers,
// so the Datadog headers are combined into one header with the following properties:
// - Key: "_datadog"
// - Value: MessageAttributeValue object
// - DataType: "String"
// - StringValue: <JSON map with key-value headers>
if (headers.TryGetValue("_datadog", out var messageAttributeValue)
&& messageAttributeValue.StringValue is string jsonString)
{
var datadogDictionary = JsonConvert.DeserializeObject<Dictionary<string, string>>(jsonString);
if (datadogDictionary.TryGetValue(name, out string value))
{
return new[] { value };
}
}
return Enumerable.Empty<string>();
}
When using the SpanContextExtractor
API to trace Kafka consumer spans, set DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED
to false
. This ensures the consumer span is correctly closed immediately after the message is consumed from the topic, and the metadata (such as partition
and offset
) is recorded correctly. Spans created from Kafka messages using the SpanContextExtractor
API are children of the producer span, and siblings of the consumer span.
If you need to propagate trace context manually (for libraries that are not instrumented automatically, like the WCF client), you can use the SpanContextInjection
API. Here is an example for WCF where this
is the WCF client:
using (OperationContextScope ocs = new OperationContextScope(this.InnerChannel))
{
var spanContextInjector = new SpanContextInjector();
spanContextInjector.Inject(OperationContext.Current.OutgoingMessageHeaders, SetHeaderValues, Tracer.Instance.ActiveScope?.Span?.Context);
}
void SetHeaderValues(MessageHeaders headers, string name, string value)
{
MessageHeader header = MessageHeader.CreateHeader(name, "datadog", value);
headers.Add(header);
}
When the Datadog SDK is configured with the Datadog format for extraction or injection (possibly both), the Datadog SDK interacts with the following request headers:
x-datadog-trace-id
- Specifies the lower 64-bits of the 128-bit trace-id, in decimal format.
x-datadog-parent-id
- Specifies the 64-bits span-id of the current span, in decimal format.
x-datadog-origin
- Specifies the Datadog product that initiated the trace, such as Real User Monitoring or Synthetic Monitoring. If this header is present, the value is expected to be one of:
rum
, synthetics
, synthetics-browser
. x-datadog-sampling-priority
- Specifies the sampling decision made for the represented span as an integer, in decimal format.
x-datadog-tags
- Specifies supplemental Datadog trace state information, including but not limited to the higher 64-bits of the 128-bit trace-id (in hexadecimal format).
When the Datadog SDK is configured with the None format for extraction or injection (possibly both), the Datadog SDK does not interact with request headers, meaning that the corresponding context propagation operation does nothing.
Baggage
Currently available in Python and Node.js. For other languages, please reach out to Support
By default, Baggage is automatically propagated through a distributed request using OpenTelemetry’s W3C-compatible headers. To disable baggage, set DD_TRACE_PROPAGATION_STYLE to datadog,tracecontext
.
Further reading
Additional helpful documentation, links, and articles: