In most cases, header extraction and injection are automatic. However, there are some known cases where your distributed trace can be disconnected. For instance, when reading messages from a distributed queue, some libraries may lose the span context. It also happens if you set DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED to false when consuming Kafka messages. In these cases, you can add a custom trace using the following code:
var spanContextExtractor = new SpanContextExtractor();
var parentContext = spanContextExtractor.Extract(headers, (headers, key) => GetHeaderValues(headers, key));
var spanCreationSettings = new SpanCreationSettings() { Parent = parentContext };
using var scope = Tracer.Instance.StartActive("operation", spanCreationSettings);
Provide the GetHeaderValues method. The way this method is implemented depends on the structure that carries SpanContext.
Here are some examples:
// Confluent.Kafka
IEnumerable<string> GetHeaderValues(Headers headers, string name)
{
if (headers.TryGetLastBytes(name, out var bytes))
{
try
{
return new[] { Encoding.UTF8.GetString(bytes) };
}
catch (Exception)
{
// ignored
}
}
return Enumerable.Empty<string>();
}
// RabbitMQ
IEnumerable<string> GetHeaderValues(IDictionary<string, object> headers, string name)
{
if (headers.TryGetValue(name, out object value) && value is byte[] bytes)
{
return new[] { Encoding.UTF8.GetString(bytes) };
}
return Enumerable.Empty<string>();
}
// SQS
public static IEnumerable<string> GetHeaderValues(IDictionary<string, MessageAttributeValue> headers, string name)
{
// For SQS, there are a maximum of 10 message attribute headers,
// so the Datadog headers are combined into one header with the following properties:
// - Key: "_datadog"
// - Value: MessageAttributeValue object
// - DataType: "String"
// - StringValue: <JSON map with key-value headers>
if (headers.TryGetValue("_datadog", out var messageAttributeValue)
&& messageAttributeValue.StringValue is string jsonString)
{
var datadogDictionary = JsonConvert.DeserializeObject<Dictionary<string, string>>(jsonString);
if (datadogDictionary.TryGetValue(name, out string value))
{
return new[] { value };
}
}
return Enumerable.Empty<string>();
}
When using the SpanContextExtractor API to trace Kafka consumer spans, set DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED to false. This ensures the consumer span is correctly closed immediately after the message is consumed from the topic, and the metadata (such as partition and offset) is recorded correctly. Spans created from Kafka messages using the SpanContextExtractor API are children of the producer span, and siblings of the consumer span.
If you need to propagate trace context manually (for libraries that are not instrumented automatically, like the WCF client), you can use the SpanContextInjection API. Here is an example for WCF where this is the WCF client:
using (OperationContextScope ocs = new OperationContextScope(this.InnerChannel))
{
var spanContextInjector = new SpanContextInjector();
spanContextInjector.Inject(OperationContext.Current.OutgoingMessageHeaders, SetHeaderValues, Tracer.Instance.ActiveScope?.Span?.Context);
}
void SetHeaderValues(MessageHeaders headers, string name, string value)
{
MessageHeader header = MessageHeader.CreateHeader(name, "datadog", value);
headers.Add(header);
}
