The Service Map for APM is here!

Advanced APM Setup

Custom Tagging

Custom tagging allows adding tags in the form of key-value pairs to specific spans. These tags are used to correlate traces with other Datadog products to provide more details about specific spans.

Read more about tagging

Tags are key-value pairs attached to spans. All tags share a single namespace.

The Datadog UI uses specific tags to set UI properties, such as an application’s service name. A full list of these tags can be found in the Datadog and OpenTracing APIs.

Custom Tags:

Custom tags are set using the OpenTracing API.

Custom tags may be set for auto-instrumentation by grabbing the active span out of the global tracer.

import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@WebServlet
class ServletImpl extends AbstractHttpServlet {
  @Override
  void doGet(HttpServletRequest req, HttpServletResponse resp) {
    final Tracer tracer = GlobalTracer.get();
    if (tracer != null && tracer.activeSpan() != null) {
      tracer.activeSpan().setTag("customer.id", 12345);
      tracer.activeSpan().setTag("http.url", "/login");
    }
    // servlet impl
  }
}

Adding tags to a span

Add tags directly to a span by calling set_tag. For example, with the following route handler:

from ddtrace import tracer

@app.route('/customer/<int:customer_id>')
def handle_customer(customer_id):
  with tracer.trace('web.request') as span:
    span.set_tag('customer.id', customer_id)

Adding tags to a current active span

The current span can be retrieved from the context in order to set tags. This way, if a span was started by the instrumentation, you can retrieve the span and add custom tags. Note that if a span does not exist, None is returned:

from ddtrace import tracer

@app.route('/customer/<int:customer_id>')
@tracer.wrap()
def handle_customer(customer_id):
  # get the active span in the context, put there by tracer.wrap()
  current_span = tracer.current_span()
  if current_span:
    current_span.set_tag('customer.id', customer_id)

Adding tags globally to all spans

Add tags to all spans by configuring the tracer with the tracer.set_tags method:

from ddtrace import tracer

tracer.set_tags({ 'env': 'prod' })

Adding tags to a span

Add tags directly to Datadog::Span objects by calling #set_tag:

# An example of a Sinatra endpoint,
# with Datadog tracing around the request.
get '/posts' do
  Datadog.tracer.trace('web.request') do |span|
    span.set_tag('http.url', request.path)
  end
end

Adding tags to a current active span

Access the current active span from any method within your code. Note, however, that if the method is called and there is no span currently active, active_span will be nil.

# e.g. adding tag to active span

current_span = Datadog.tracer.active_span
current_span.set_tag('<TAG_KEY>', '<TAG_VALUE>') unless current_span.nil?

Adding tags globally to all spans

Add tags to all spans by configuring the tracer with the tags option:

Datadog.configure do |c|
  c.tracer tags: { 'env' => 'prod' }
end

See the API documentation for more details.

Adding tags to a span

Add tags directly to a Span interface by calling SetTag:

package main

import (
    "log"
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Set tag
    span.SetTag("http.url", r.URL.Path)
}

func main() {
    tracer.Start(tracer.WithServiceName("<SERVICE_NAME>"))
    defer tracer.Stop()
    http.HandleFunc("/posts", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Adding tags to a Span attached to a Context

Our integrations make use of the Context type to propagate the current active span. If you want to add a tag to a span attached to a Context via automatic instrumentation, call the SpanFromContext function:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Retrieve a span for a web request attached to a Go Context.
    if span, ok := tracer.SpanFromContext(r.Context()); ok {
        // Set tag
        span.SetTag("http.url", r.URL.Path)
    }
}

Adding tags globally to all spans

Add tags to all spans by configuring the tracer with the tags option:

package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start(
        tracer.WithGlobalTag("datacenter", "us-1"),
        tracer.WithGlobalTag("env", "prod"),
    )
    defer tracer.Stop()
}

Adding tags to a span

Add tags directly to span objects by calling setTag or addTags:

// An example of an Express endpoint,
// with Datadog tracing around the request.
app.get('/posts', (req, res) => {
  const span = tracer.startSpan('web.request')

  span.setTag('http.url', req.url)
  span.addTags({
    'http.method': req.method
  })
})

Adding tags to a current active span

Access the current active span from any method within your code. Note, however, that if the method is called and there is no span currently active, tracer.scopeManager().active() returns null.

// e.g. adding tag to active span

const scope = tracer.scopeManager().active()
const span = scope.span()

span.setTag('<TAG_KEY>', '<TAG_VALUE>')

Adding tags to a span

Add tags directly to a Datadog.Trace.Span object by calling Span.SetTag(). For example:

using Datadog.Trace;

// get the global tracer
var tracer = Tracer.Instance;

// get the currently active span (can be null)
var span = tracer.ActiveScope?.Span;

// add a tag to the span
span?.SetTag("<TAG_KEY>", "<TAG_VALUE>");

Note: Datadog.Trace.Tracer.Instance.ActiveScope returns null if there is no active span.

Manual Instrumentation

Manual instrumentation allows programmatic creation of traces to send to Datadog. This is useful for tracing in-house code not captured by automatic instrumentation. Before instrumenting your application, review Datadog’s APM Terminology and familiarize yourself with the core concepts of Datadog APM.

If you aren’t using a supported framework instrumentation, or you would like additional depth in your application’s traces, you may want to manually instrument your code.

Do this either using the Trace annotation for simple method call tracing, or with the OpenTracing API for complex tracing.

Datadog’s Trace annotation is provided by the dd-trace-api dependency.

Example Usage

import datadog.trace.api.Trace;

public class MyClass {
  @Trace
  public static void myMethod() {
    // your method implementation here
  }
}

If you aren’t using supported library instrumentation (see library compatibility), you may want to manually instrument your code.

You may also want to extend the functionality of the ddtrace library or gain finer control over instrumenting your application. Several techniques are provided by the library to accomplish this.

The following examples use the global tracer object which can be imported via:

  from ddtrace import tracer

Decorator

ddtrace provides a decorator that can be used to trace a particular method in your application:

  @tracer.wrap()
  def business_logic():
    """A method that would be of interest to trace."""
    # ...
    # ...

API details for the decorator can be found at ddtrace.Tracer.wrap()

Context Manager

To trace an arbitrary block of code, you can use the ddtrace.Span context manager:

  # trace some interesting operation
  with tracer.trace('interesting.operations'):
    # do some interesting operation(s)
    # ...
    # ...

Further API details can be found at ddtrace.Tracer()

Using the API

If the above methods are still not enough to satisfy your tracing needs, a manual API is provided which will allow you to start and finish spans however you may require:

  span = tracer.trace('operations.of.interest')

  # do some operation(s) of interest in between

  # NOTE: make sure to call span.finish() or the entire trace will not be sent
  # to Datadog
  span.finish()

API details of the decorator can be found here:

If you aren’t using supported library instrumentation (see library compatibility), you may want to to manually instrument your code. Adding tracing to your code is easy using the Datadog.tracer.trace method, which you can wrap around any Ruby code.

Example Usage

# An example of a Sinatra endpoint,
# with Datadog tracing around the request,
# database query, and rendering steps.
get '/posts' do
  Datadog.tracer.trace('web.request', service: '<SERVICE_NAME>', resource: 'GET /posts') do |span|
    # Trace the activerecord call
    Datadog.tracer.trace('posts.fetch') do
      @posts = Posts.order(created_at: :desc).limit(10)
    end

    # Add some APM tags
    span.set_tag('http.method', request.request_method)
    span.set_tag('posts.count', @posts.length)

    # Trace the template rendering
    Datadog.tracer.trace('template.render') do
      erb :index
    end
  end
end

For more details about manual instrumentation, check out the API documentation.

If you aren’t using supported library instrumentation (see Library compatibility), you may want to to manually instrument your code.

To make use of manual instrumentation, use the tracer package which is documented on Datadog’s godoc page.

Example Usage

package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    // Start the tracer with zero or more options.
    tracer.Start(tracer.WithServiceName("<SERVICE_NAME>"))
    defer tracer.Stop()

    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Set metadata
    span.SetTag("<TAG_KEY>", "<TAG_VALUE>")
}

If you aren’t using supported library instrumentation (see Library compatibility), you may want to manually instrument your code.

The following example initializes a Datadog Tracer and creates a span called web.request:

const tracer = require('dd-trace').init()
const span = tracer.startSpan('web.request')

span.setTag('http.url', '/login')
span.finish()

For more information on manual instrumentation, see the API documentation.

If you aren’t using libraries supported by automatic instrumentation (see Library compatibility), you should manually instrument your code.

The following example uses the global Datadog Tracer and creates a span to trace a web request:

using Datadog.Trace;

using(var scope = Tracer.Instance.StartActive("web.request"))
{
    var span = scope.Span;
    span.Type = SpanTypes.Web;
    span.ResourceName = request.Url;
    span.SetTag(Tags.HttpMethod, request.Method);

    // do some work...
}

OpenTracing

OpenTracing is a vendor-neutral, cross-language standard for tracing applications. Datadog offers OpenTracing implementations for many APM tracers. For more details see opentracing.io.

Use the OpenTracing API and the Datadog Tracer (dd-trace-ot) library to measure execution times for specific pieces of code. This lets you trace your application more precisely than you can with the Java Agent alone.

Setup:

For Maven, add this to pom.xml:

<!-- OpenTracing API -->
<dependency>
    <groupId>io.opentracing</groupId>
    <artifactId>opentracing-api</artifactId>
    <version>0.31.0</version>
</dependency>

<!-- OpenTracing Util -->
<dependency>
    <groupId>io.opentracing</groupId>
    <artifactId>opentracing-util</artifactId>
    <version>0.31.0</version>
</dependency>

<!-- Datadog Tracer (only needed if you do not use dd-java-agent) -->
<dependency>
    <groupId>com.datadoghq</groupId>
    <artifactId>dd-trace-ot</artifactId>
    <version>${dd-trace-java.version}</version>
</dependency>

For Gradle, add:

compile group: 'io.opentracing', name: 'opentracing-api', version: "0.31.0"
compile group: 'io.opentracing', name: 'opentracing-util', version: "0.31.0"
compile group: 'com.datadoghq', name: 'dd-trace-ot', version: "${dd-trace-java.version}"

Configure your application using environment variables or system properties as discussed in the configuration section.

Manual Instrumentation with OpenTracing:

Use a combination of these if the automatic instrumentation isn’t providing you enough depth or detail.

Using try-finally:

import datadog.trace.api.DDTags;

import io.opentracing.Scope;
import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

class InstrumentedClass {

    void method0() {
        /*
         * 1. Configure your application using environment variables or system properties
         * 2. Using dd-java-agent (-javaagent:/path/to/dd-java-agent.jar),
         *    GlobalTracer is automatically instantiated.
         */
        Tracer tracer = GlobalTracer.get();

        Scope scope = tracer.buildSpan("<OPERATION_NAME>").startActive(true);
        try {
            scope.span().setTag(DDTags.SERVICE_NAME, "<SERVICE_NAME>");

            // The code you're tracing
            Thread.sleep(1000);

        // If you don't call close(), the span data will NOT make it to Datadog!
        } finally {
            scope.close();
        }
    }
}

Alternatively, wrap the code you want to trace in a try-with-resources statement:

import datadog.trace.api.DDTags;

import io.opentracing.Scope;
import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

class InstrumentedClass {

    void method0() {
        Tracer tracer = GlobalTracer.get();

        try (Scope scope = tracer.buildSpan("<OPERATION_NAME>").startActive(true)) {
            scope.span().setTag(DDTags.SERVICE_NAME, "<SERVICE_NAME>");
            Thread.sleep(1000);
        }
    }
}

In this case, you don’t need to call scope.close().

If you’re not using dd-java-agent.jar, you must register a configured tracer with GlobalTracer. For this, call GlobalTracer.register(new DDTracer()) early on in your application startup (e.g., main method).

import datadog.opentracing.DDTracer;
import datadog.trace.api.sampling.AllSampler;
import datadog.trace.common.writer.DDAgentWriter;

import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

public class Application {

    public static void main(String[] args) {

        // Initialize the tracer from environment variables or system properties
        Tracer tracer = new DDTracer();
        GlobalTracer.register(tracer);
        // register the same tracer with the Datadog API
        datadog.trace.api.GlobalTracer.registerIfAbsent(tracer);

        // OR from the API
        Writer writer = new DDAgentWriter();
        Sampler sampler = new AllSampler();
        Tracer tracer = new DDTracer(writer, sampler);
        GlobalTracer.register(tracer);
        // register the same tracer with the Datadog API
        datadog.trace.api.GlobalTracer.registerIfAbsent(tracer);

        // ...
    }
}

Manual Instrumentation for Async Traces:

Create asynchronous traces with manual instrumentation using the OpenTracing API.

// Step 1: start the Scope/Span on the work submission thread
try (Scope scope = tracer.buildSpan("ServiceHandlerSpan").startActive(false)) {
    final Span span = scope.span();
    doAsyncWork(new Runnable() {
        @Override
        public void run() {
            // Step 2: reactivate the Span in the worker thread
            try (Scope scope = tracer.scopeManager().activate(span, false)) {
              // worker thread impl...
            }
        }
    });
    // submission thread impl...
}

Notice the above examples only use the OpenTracing classes. Check the OpenTracing API for more details and information.

Support for OpenTracing in the Python tracer is currently in beta.

Setup:

OpenTracing support is included in the ddtrace package. Use pip to install the required opentracing package :

$ pip install ddtrace[opentracing]

Usage:

The OpenTracing convention for initializing a tracer is to define an initialization method that configures and instantiates a new tracer and overwrites the global opentracing.tracer reference:

import time
import opentracing
from ddtrace.opentracer import Tracer, set_global_tracer

def init_tracer(service_name):
    config = {
      'agent_hostname': 'localhost',
      'agent_port': 8126,
    }
    tracer = Tracer(service_name, config=config)
    set_global_tracer(tracer)
    return tracer

def my_operation():
  span = opentracing.tracer.start_span('<OPERATION_NAME>')
  span.set_tag('<TAG_KEY>', '<TAG_VALUE>')
  time.sleep(0.05)
  span.finish()

init_tracer('<SERVICE_NAME>')
my_operation()

For more advanced usage and configuration information see Datadog Python Opentracing API docs and the Python OpenTracing repo.

Support for OpenTracing with Ruby is coming soon. Reach out to the Datadog support team to be part of the beta.

Import the opentracer package to expose the Datadog tracer as an OpenTracing compatible tracer.

Example:

A basic usage example:

package main

import (
    "github.com/opentracing/opentracing-go"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/opentracer"
    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func main() {
    // Start the regular tracer and return it as an opentracing.Tracer interface. You
    // may use the same set of options as you normally would with the Datadog tracer.
    t := opentracer.New(tracer.WithServiceName("<SERVICE_NAME>"))

    // Stop it using the regular Stop call for the tracer package.
    defer tracer.Stop()

    // Set the global OpenTracing tracer.
    opentracing.SetGlobalTracer(t)

    // Use the OpenTracing API as usual.
}

Note: Using the OpenTracing API in parallel with the regular API or Datadog integrations is fully supported. Under the hood, all of them make use of the same tracer. See the API documentation for more examples and details.

This library is OpenTracing compliant. Use the (OpenTracing API and the Datadog Tracer (dd-trace) library to measure execution times for specific pieces of code. In the following example, a Datadog Tracer is initialized and used as a global tracer:

// server.js

const tracer = require('dd-trace').init()
const opentracing = require('opentracing')

opentracing.initGlobalTracer(tracer)

const app = require('./app.js')

// app.js

const tracer = opentracing.globalTracer()

The following tags are available to override Datadog specific options:

  • service.name: The service name to be used for this span. The service name from the tracer will be used if this is not provided.
  • resource.name: The resource name to be used for this span. The operation name will be used if this is not provided.
  • span.type: The span type to be used for this span. Will fallback to custom if not provided.

For OpenTracing support, add the Datadog.Trace.OpenTracing NuGet package to your application. During application start-up, initialize the OpenTracing library:

public void ConfigureServices(IServiceCollection services)
{
    // create an OpenTracing ITracer with default setting
    OpenTracing.ITracer tracer =
        Datadog.Trace.OpenTracing.OpenTracingTracerFactory.CreateTracer();
    
    // to use tracer with ASP.NET Core dependency injection
    services.AddSingleton<ITracer>(tracer);

    // to use tracer with OpenTracing.GlobalTracer.Instance
    GlobalTracer.Register(tracer);
}

Distributed Tracing

Distributed tracing allows you to propagate a single trace across multiple services and hosts, so you can see performance end-to-end. Linking is implemented by injecting Datadog Metadata into the request headers.

Distributed tracing headers are language agnostic. A trace started in one language may propagate to another (for example, from Python to Java).

Distributed traces may sample inconsistently when the linked traces run on different hosts. To ensure that distributed traces are complete, enable priority sampling.

Create a distributed trace using manual instrumentation with OpenTracing:

// Step 1: Inject the Datadog headers in the client code
try (Scope scope = tracer.buildSpan("httpClientSpan").startActive(true)) {
    final Span span = scope.span();
    HttpRequest request = /* your code here */;

    tracer.inject(span.context(),
                  Format.Builtin.HTTP_HEADERS,
                  new MyHttpHeadersInjectAdapter(request));

    // http request impl...
}

public static class MyHttpHeadersInjectAdapter implements TextMap {
  private final HttpRequest httpRequest;

  public HttpHeadersInjectAdapter(final HttpRequest httpRequest) {
    this.httpRequest = httpRequest;
  }

  @Override
  public void put(final String key, final String value) {
    httpRequest.addHeader(key, value);
  }

  @Override
  public Iterator<Map.Entry<String, String>> iterator() {
    throw new UnsupportedOperationException("This class should be used only with tracer#inject()");
  }
}

// Step 2: Extract the Datadog headers in the server code
HttpRequest request = /* your code here */;

final SpanContext extractedContext =
  GlobalTracer.get().extract(Format.Builtin.HTTP_HEADERS,
                             new MyHttpRequestExtractAdapter(request));

try (Scope scope = tracer.buildSpan("httpServerSpan").asChildOf(extractedContext).startActive(true)) {
    final Span span = scope.span(); // will be a child of http client span in step 1
    // http server impl...
}

public class MyHttpRequestExtractAdapter implements TextMap {
  private final HttpRequest request;

  public HttpRequestExtractAdapter(final HttpRequest request) {
    this.request = request;
  }

  @Override
  public Iterator<Map.Entry<String, String>> iterator() {
    return request.headers().iterator();
  }

  @Override
  public void put(final String key, final String value) {
    throw new UnsupportedOperationException("This class should be used only with Tracer.extract()!");
  }
}

Create a distributed trace by manually propagating the tracing context:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    span, ctx := tracer.StartSpanFromContext(r.Context(), "post.process")
    req, err := http.NewRequest("GET", "http://example.com", nil)
    req = req.WithContext(ctx)
    // Inject the span Context in the Request headers
    err = tracer.Inject(span.Context(), tracer.HTTPHeadersCarrier(r.Header))
    if err != nil {
        // Handle or log injection error
    }
    http.DefaultClient.Do(req)
}

Then, on the server side, to continue the trace, start a new Span from the extracted Context:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Extract the span Context and continue the trace in this service
    sctx, err := tracer.Extract(tracer.HTTPHeadersCarrier(r.Header))
    if err != nil {
        // Handle or log extraction error
    }

    span := tracer.StartSpan("post.filter", tracer.ChildOf(sctx))
    defer span.Finish()
}

Distributed tracing is enabled by default for all supported integrations (see Compatibility).

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Priority Sampling

Priority sampling allows traces between two Datadog endpoints to be sampled together. This prevents trace sampling from removing segments of a distributed trace (i.e. ensures completeness). Additionally, APM traces expose sampling flags to configure how specific traces are sampled.

Priority sampling automatically assigns and propagates a priority value along all traces, depending on their service and volume. Priorities can also be set manually to drop non-interesting traces or keep important ones.

For a more detailed explanations of sampling and priority sampling, check the sampling and storage documentation.

Priority sampling is disabled by default. To enable it, configure the priority.sampling flag to true (see how to configure the java client.

Current Priority Values (more may be added in the future):

Sampling Value Effect
SAMPLER_DROP The sampler automatically decided to not keep the trace. The Agent will drop it.
SAMPLER_KEEP The sampler automatically decided to keep the trace. The Agent will keep it. Might be sampled server-side.
USER_DROP The user asked to not keep the trace. The Agent will drop it.
USER_KEEP The user asked to keep the trace. The Agent will keep it. The server will keep it too.

Manually set trace priority:

import datadog.trace.api.Trace;
import datadog.trace.api.interceptor.MutableSpan;
import datadog.trace.common.sampling.PrioritySampling;
import io.opentracing.util.GlobalTracer;

public class MyClass {
    @Trace
    public static void myMethod() {
        // grab the active span out of the traced method
        MutableSpan ddspan = (MutableSpan) GlobalTracer.get().activeSpan();
        // ask the sampler to keep the current trace
        ddspan.setSamplingPriority(PrioritySampling.USER_KEEP);
        // method impl follows
    }
}

Priority sampling is disabled by default. To enable it, configure the priority_sampling flag using the tracer.configure method:

tracer.configure(priority_sampling=True)

To set a custom priority to a trace:

from ddtrace.ext.priority import USER_REJECT, USER_KEEP

span = tracer.current_span()

# indicate to not keep the trace
span.context.sampling_priority = USER_REJECT

The following priorities can be used.

Sampling Value Effect
AUTO_REJECT The sampler automatically decided to not keep the trace. The Agent will drop it.
AUTO_KEEP The sampler automatically decided to keep the trace. The Agent will keep it. Might be sampled server-side.
USER_REJECT The user asked to not keep the trace. The Agent will drop it.
USER_KEEP The user asked to keep the trace. The Agent will keep it. The server will keep it too.

Priority sampling is disabled by default. Enabling it ensures that your sampled distributed traces will be complete. To enable the priority sampling:

Datadog.configure do |c|
  c.tracer priority_sampling: true
end

Once enabled, the sampler automatically assigns a value of AUTO_REJECT or AUTO_KEEP to traces, depending on their service and volume.

You can also set this priority manually to either drop a non-interesting trace or to keep an important one. For that, set the Context#sampling_priority to:

# To reject the trace
span.context.sampling_priority = Datadog::Ext::Priority::USER_REJECT

# To keep the trace
span.context.sampling_priority = Datadog::Ext::Priority::USER_KEEP

Possible values for the sampling priority tag are:

Sampling Value Effect
Datadog::Ext::Priority::AUTO_REJECT The sampler automatically decided to not keep the trace. The Agent will drop it.
Datadog::Ext::Priority::AUTO_KEEP The sampler automatically decided to keep the trace. The Agent will keep it. Might be sampled server-side.
Datadog::Ext::Priority::USER_REJECT The user asked to not keep the trace. The Agent will drop it.
Datadog::Ext::Priority::USER_KEEP The user asked to keep the trace. The Agent will keep it. The server will keep it too.

When not using distributed tracing, you may change the priority at any time, as long as the trace is not finished yet. However, it must be done before any context propagation (e.g. fork, RPC calls) to be effective in a distributed context. Changing the priority after such context has been propagated causes different parts of a distributed trace to use different priorities. Some parts might be kept, some parts might be rejected, and consequently this can cause the trace to be partially stored and remain incomplete.

It is recommended that changing priority be done as soon as possible, when the root span has just been created.

See the API documentation for more details.

For more details about how to use and configure distributed tracing, check out the godoc page.

Set the sampling priority of a trace by adding the sampling.priority tag to its root span. This is then propagated throughout the entire stack. For example:

package main

import (
    "log"
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/ext"
    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Set priority sampling as a regular tag
    span.SetTag(ext.SamplingPriority, ext.PriorityUserKeep)
}

Possible values for the sampling priority tag are:

Sampling Value Effect
ext.PriorityAutoReject The sampler automatically decided to not keep the trace. The Agent will drop it.
ext.PriorityAutoKeep The sampler automatically decided to keep the trace. The Agent will keep it. Might be sampled server-side.
ext.PriorityUserReject The user asked to not keep the trace. The Agent will drop it.
ext.PriorityUserKeep The user asked to keep the trace. The Agent will keep it. The server will keep it too.

Priority sampling is enabled by default. The sampler automatically assigns a value of AUTO_REJECT or AUTO_KEEP to traces, depending on their service and volume.

Set this priority manually to either drop a non-interesting trace or to keep an important one with the sampling.priority tag:

const priority = require('dd-trace/ext/priority')

// To reject the trace
span.setTag('sampling.priority', priority.USER_REJECT)

// To keep the trace
span.setTag('sampling.priority', priority.USER_KEEP)

Possible values for the sampling priority tag are:

Sampling Value Effect
AUTO_REJECT The sampler automatically decided to not keep the trace. The Agent will drop it.
AUTO_KEEP The sampler automatically decided to keep the trace. The Agent will keep it. Might be sampled server-side.
USER_REJECT The user asked to not keep the trace. The Agent will drop it.
USER_KEEP The user asked to keep the trace. The Agent will keep it. The server will keep it too.

Once the sampling priority has been set, it cannot be changed. This is done automatically whenever a span is finished or the trace is propagated. Setting it manually should thus be done before either occur.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Logging

Datadog’s logging APIs allow for accessing active tracing identifiers, which can be used to correlate APM traces with specific log events.

The Java tracer exposes two API calls to allow printing trace and span identifiers along with log statements, CorrelationIdentifier#getTraceId(), and CorrelationIdentifier#getSpanId().

These identifiers can be injected into application logs using MDC frameworks.

log4j2:

import org.apache.logging.log4j.ThreadContext;
import datadog.trace.api.CorrelationIdentifier;

// there must be spans started and active before this block.
try {
    ThreadContext.put("ddTraceID", "ddTraceID:" + String.valueOf(CorrelationIdentifier.getTraceId()));
    ThreadContext.put("ddSpanID", "ddSpanID:" + String.valueOf(CorrelationIdentifier.getSpanId()));
} finally {
    ThreadContext.remove("ddTraceID");
    ThreadContext.remove("ddSpanID");
}

slf4j/logback:

import org.slf4j.MDC;
import datadog.trace.api.CorrelationIdentifier;

// there must be spans started and active before this block.
try {
    MDC.put("ddTraceID", "ddTraceID:" + String.valueOf(CorrelationIdentifier.getTraceId()));
    MDC.put("ddSpanID", "ddSpanID:" + String.valueOf(CorrelationIdentifier.getSpanId()));
} finally {
    MDC.remove("ddTraceID");
    MDC.remove("ddSpanID");
}

log4j2 XML Pattern:

<PatternLayout pattern="%clr{%d{yyyy-MM-dd HH:mm:ss.SSS}}{faint} %clr{%5p} %clr{${sys:PID}}{magenta} %clr{---}{faint} %X{ddTraceID} %X{ddSpanID} %m%n%xwEx" />

Logback XML Pattern:

<Pattern>
    %d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} %X{ddTraceID} %X{ddSpanID} - %msg%n
</Pattern>

Getting the required information needed for logging is easy:

from ddtrace import helpers

trace_id, span_id = helpers.get_correlation_ids()

Coming Soon. Reach out to the Datadog support team to be part of the beta.

The Go tracer exposes two API calls to allow printing trace and span identifiers along with log statements using exported methods from SpanContext type:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Retrieve Trace ID and Span ID
    traceID := span.Context().TraceID()
    spanID := span.Context().SpanID()
}

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Coming Soon. Reach out to the Datadog support team to be part of the beta.

Debugging

Datadog debug settings are used to diagnose issues or audit trace data.

We discourage enabling debug mode on your production systems as it increases the number of events that are sent to your loggers. Use sparingly for debugging purposes only.

To return debug level application logs, enable debug mode with the flag -Ddatadog.slf4j.simpleLogger.defaultLogLevel=debug when starting the JVM.

Debugging is disabled by default.

To enable it set the environment variable DATADOG_TRACE_DEBUG=true when using ddtrace-run.

Debug mode is disabled by default. To enable:

Datadog.configure do |c|
  c.tracer debug: true
end

Application Logs:

By default, all logs are processed by the default Ruby logger. When using Rails, you should see the messages in your application log file.

Datadog client log messages are marked with [ddtrace], so you can isolate them from other messages.

Additionally, it is possible to override the default logger and replace it with a custom one. This is done using the log attribute of the tracer.

f = File.new("<FILENAME>.log", "w+")           # Log messages should go there
Datadog.configure do |c|
  c.tracer log: Logger.new(f)                 # Overriding the default tracer
end

Datadog::Tracer.log.info { "this is typically called by tracing code" }

See the API documentation for more details.

Debug mode on the tracer can be enabled as a Start config, resulting in more verbose logging:

package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start(tracer.WithDebugMode(true))
    defer tracer.Stop()
}

Debug mode is disabled by default, to enable it:

const tracer = require('dd-trace').init({
  debug: true
})

Application Logs:

By default, logging from this library is disabled. In order to get debbuging information and errors sent to logs, the debug options should be set to true in the init() method.

The tracer will then log debug information to console.log() and errors to console.error(). This behavior can be changed by passing a custom logger to the tracer. The logger should contain debug() and error() methods that can handle messages and errors, respectively.

For example:

const bunyan = require('bunyan')
const logger = bunyan.createLogger({
  name: 'dd-trace',
  level: 'trace'
})

const tracer = require('dd-trace').init({
  logger: {
    debug: message => logger.trace(message),
    error: err => logger.error(err)
  },
  debug: true
})

For more tracer settings, check out the API documentation.

Debug mode is disabled by default. To enable it, set the isDebugEnabled argument to true when creating a new tracer instance:

var tracer = Datadog.Trace.Tracer.Create(isDebugEnabled: true);

Security

Sensitive information within your traces can be scrubbed automatically or manually.

Automatic scrubbing

Automatic scrubbing is available for some services, such as ElasticSearch, MongoDB, Redis, Memcached, and HTTP server and client request URLs. Below is an example configuration snippet documenting all the available options.

apm_config:
  # Defines obfuscation rules for sensitive data. Disabled by default.
  obfuscation:
    # ElasticSearch obfuscation rules. Applies to spans of type "elasticsearch".
    # More specifically, to the "elasticsearch.body" tag.
    elasticsearch:
      enabled: true
      # Values for the keys listed here will not be obfuscated.
      keep_values:
        - client_id
        - product_id

    # MongoDB obfuscation rules. Applies to spans of type "mongodb".
    # More specifically, to the "mongodb.query" tag.
    mongodb:
      enabled: true
      # Values for the keys listed here will not be obfuscated.
      keep_values:
        - document_id
        - template_id

    # HTTP obfuscation rules for "http.url" tags in spans of type "http".
    http:
      # If true, query strings in URLs will be obfuscated.
      remove_query_string: true
      # If true, path segments in URLs containing digits will be replaced by "?"
      remove_paths_with_digits: true

    # When enabled, stack traces will be removed (replaced by "?").
    remove_stack_traces: true

    # Obfuscation rules for spans of type "redis". Applies to the "redis.raw_command" tags.
    redis:
      enabled: true

    # Obfuscation rules for spans of type "memcached". Applies to the "memcached.command" tag.
    memcached:
      enabled: true

Replace rules

To scrub sensitive data from your span’s tags, use the replace_tags setting. It is a list containing one or more groups of parameters that describe how to perform replacements of sensitive data within your tags. These parameters are:

  • name: The key of the tag to replace. To match all tags, use *. To match the resource, use resource.name.
  • pattern: The regexp pattern to match against.
  • repl: The replacement string.

For example:

apm_config:
  replace_tags:
    # Replace all numbers following the `token/` string in the tag "http.url" with "?":
    - name: "http.url"
      pattern: "token/(.*)"
      repl: "?"
    # Replace all the occurrences of "foo" in any tag with "bar":
    - name: "*"
      pattern: "foo"
      repl: "bar"
    # Remove all "error.stack" tag's value.
    - name: "error.stack"
      pattern: "(?s).*"