If you have not read the setup instructions for automatic instrumentation, start with the Python Setup Instructions.

If you aren't using supported library instrumentation (see library compatibility), you may want to manually instrument your code.

You may also want to extend the functionality of the ddtrace library or gain finer control over instrumenting your application. Several techniques are provided by the library to accomplish this.

Creating spans

The ddtrace library creates spans automatically with ddtrace-run for many libraries and frameworks. However, you may want to gain visibility into your own code and this is achieved by using spans.

Within your web request (for example, make_sandwich_request), you may perform several operations, like get_ingredients() and assemble_sandwich(), which are useful to measure.

def make_sandwich_request(request):
    ingredients = get_ingredients()
    sandwich = assemble_sandwich(ingredients)

Using decorators

ddtrace provides a decorator tracer.wrap() that can be used to decorate the functions of interest. This is useful if you would like to trace the function regardless of where it is being called from.

from ddtrace import tracer

@tracer.wrap(service="my-sandwich-making-svc", resource="resource_name")
def get_ingredients():
    # go to the pantry
    # go to the fridge
    # maybe go to the store
    return

# You can provide more information to customize the span
@tracer.wrap("assemble_sandwich", service="my-sandwich-making-svc", resource="resource_name")
def assemble_sandwich(ingredients):
    return

To learn more, read API details for the decorator for ddtrace.Tracer.wrap().

Using context managers

To trace an arbitrary block of code, use the ddtrace.Span context manager as below, or view the advanced usage documentation.

from ddtrace import tracer

def make_sandwich_request(request):
    # Capture both operations in a span
    with tracer.trace("sandwich.make"):
        ingredients = get_ingredients()
        sandwich = assemble_sandwich(ingredients)

def make_sandwich_request(request):
    # Capture both operations in a span
    with tracer.trace("sandwich.create", resource="resource_name") as outer_span:

        with tracer.trace("get_ingredients", resource="resource_name") as span:
            ingredients = get_ingredients()

        with tracer.trace("assemble_sandwich", resource="resource_name") as span:
            sandwich = assemble_sandwich(ingredients)

To learn more, read the full API details for ddtrace.Tracer().

Manual span creation

If the decorator and context manager methods are still not enough to satisfy your tracing needs, a manual API is provided which allows you to start and finish spans however you may require:

def make_sandwich_request(request):
    span = tracer.trace("sandwich.create", resource="resource_name")
    ingredients = get_ingredients()
    sandwich = assemble_sandwich(ingredients)
    span.finish()  # remember to finish the span

For more API details of the decorator, read the ddtrace.Tracer.trace documentation or the ddtrace.Span.finish documentation.

Accessing active spans

The built-in instrumentation and your own custom instrumentation create spans around meaningful operations. You can access the active span in order to include meaningful data.

from ddtrace import tracer

def make_sandwich_request(request):
    # Capture both operations in a span
    with tracer.trace("sandwich.make") as my_span:
        ingredients = get_ingredients()
        sandwich = assemble_sandwich(ingredients)

Current span

def get_ingredients():
    # Get the active span
    span = tracer.current_span()
    # this span is my_span from make_sandwich_request above

Root span

def assemble_sandwich(ingredients):
    with tracer.trace("another.operation") as another_span:
        # Get the active root span
        span = tracer.current_root_span()
        # this span is my_span from make_sandwich_request above

Adding tags

Adding tags locally

Tags can be added to a span using the set_tag method on a span:

from ddtrace import tracer

def make_sandwich_request(request):
    with tracer.trace("sandwich.make") as span:
        ingredients = get_ingredients()
        span.set_tag("num_ingredients", len(ingredients))

Adding tags globally

Tags can be globally set on the tracer. These tags are be applied to every span that is created.

from ddtrace import tracer
from myapp import __version__

# This will be applied to every span
tracer.set_tags({"version": __version__, "<TAG_KEY_2>": "<TAG_VALUE_2>"})

Setting errors on a span

Exception information is captured and attached to a span if there is one active when the exception is raised.

from ddtrace import tracer

with tracer.trace("throws.an.error") as span:
    raise Exception("Oops!")

# `span` will be flagged as erroneous and have
# the stack trace and exception message attached as tags

Flagging a span as erroneous can also be done manually:

from ddtrace import tracer

span = tracer.trace("operation")
span.error = 1
span.finish()

In the event you want to flag the local root span with the error raised:

import os
from ddtrace import tracer

try:
    raise TypeError
except TypeError as e:
    root_span = tracer.current_root_span()
    (exc_type, exc_val, exc_tb) = sys.exc_info()
    # this sets the error type, marks the span as an error, and adds the traceback
    root_span.set_exc_info(exc_type, exc_val, exc_tb)

Propagating context with headers extraction and injection

You can configure the propagation of context for distributed traces by injecting and extracting headers. Read Trace Context Propagation for information.

Baggage

Manipulating Baggage on a span:

from ddtrace import tracer

# Start a new span and set baggage
with tracer.trace("example") as span:
    # set_baggage_item
    span.context.set_baggage_item("key1", "value1")
    span.context.set_baggage_item("key2", "value2")

    # get_all_baggage_items
    all_baggage = span.context.get_all_baggage_items()
    print(all_baggage) # {'key1': 'value1', 'key2': 'value2'}

    # remove_baggage_item
    span.context.remove_baggage_item("key1")
    print(span.context.get_all_baggage_items()) # {'key2': 'value2'}

    # get_baggage_item
    print(span.context.get_baggage_item("key1")) # None
    print(span.context.get_baggage_item("key2")) # value2

    # remove_all_baggage_items
    span.context.remove_all_baggage_items()
    print(span.context.get_all_baggage_items()) # {}

To see an example in action, see flask-baggage on trace-examples.

ddtrace-api

The ddtrace-api package provides a stable public API for Datadog APM's custom Python instrumentation. This package implements only the API interface, not the underlying functionality that creates and sends spans to Datadog.

This separation between interface (ddtrace-api) and implementation (ddtrace) offers several benefits:

• You can rely on an API that changes less frequently and more predictably for your custom instrumentation
• If you only use automatic instrumentation, you can ignore API changes entirely
• If you implement both single-step and custom instrumentation, you avoid depending on multiple copies of the ddtrace package

To use ddtrace-api:

1. Install both the ddtrace and ddtrace-api libraries:

   Copy

   pip install 'ddtrace>=3.1' ddtrace-api
   

2. Instrument your Python application using ddtrace-run by prefixing your Python entry-point command:

   Copy

   ddtrace-run python app.py
   

3. After this is set up, you can write custom instrumentation exactly like the examples in the previous sections, but you import from ddtrace_api instead of ddtrace.

   For example:

   Copy

   from ddtrace_api import tracer
   
   @tracer.wrap(service="my-sandwich-making-svc", resource="resource_name")
   def get_ingredients():
       # go to the pantry
       # go to the fridge
       # maybe go to the store
       return
   

See that package's API definition for the full list of supported API calls.

If you aren't using supported library instrumentation (see library compatibility), you may want to manually instrument your code.

You may also want to extend the functionality of the dd-trace library or gain finer control over instrumenting your application. Several techniques are provided by the library to accomplish this.

Adding tags

The built-in instrumentation and your own custom instrumentation create spans around meaningful operations.

Adding tags locally

You can access the active span in order to include meaningful data by adding tags.

const span = tracer.scope().active()

To learn more, read API details for Scope.

You can add tags to a span using the setTag or addTags method on a span. Supported value types are string, number, and object.

// add a foo:bar tag
span.setTag('foo', 'bar')

// add a user_id:5 tag
span.setTag('user_id', 5)

// add a obj.first:foo and obj.second:bar tags
span.setTag('obj', { first: 'foo', second: 'bar' })

// add a foo:bar and baz:qux tags
span.addTags({
  foo: 'bar',
  baz: 'qux'
})

Adding tags globally

You can add tags to every span by configuring them directly on the tracer, either with the comma-separated DD_TAGS environment variable or with the tags option on the tracer initialization:

// equivalent to DD_TAGS=foo:bar,baz:qux
tracer.init({
  tags: {
    foo: 'bar',
    baz: 'qux'
  }
})

// All spans will now have these tags

Adding tags through component hooks

Some Datadog integrations support span hooks that can be used to update the span right before it's finished. This is useful to modify or add tags to a span that is otherwise inaccessible from your code.

// at the top of the entry point right after tracer.init()
tracer.use('express', {
  // hook will be executed right before the request span is finished
  hooks: {
    request: (span, req, res) => {
      span.setTag('customer.id', req.query.customer_id)
    }
  }
})

To learn more, read API details for individual plugins.

Setting errors on a span

Errors can be added to a span with the special error tag that supports error objects. This splits the error into three tags: error.type, error.message, and error.stack.

try {
  getIngredients()
} catch (e) {
  span.setTag('error', e)
}

When using tracer.trace() or tracer.wrap() this is done automatically when an error is thrown.

Creating spans

The dd-trace library creates spans automatically with tracer.init() for many libraries and frameworks. However, you may want to gain visibility into your own code and this is achieved using spans.

Within your web request (for example, /make-sandwich), you may perform several operations, like getIngredients() and assembleSandwich(), which are useful to measure.

Synchronous code

Synchronous code can be traced with tracer.trace() which automatically finishes the span when its callback returns and captures any thrown error automatically.

app.get('/make-sandwich', (req, res) => {
  const sandwich = tracer.trace('sandwich.make', { resource: 'resource_name' }, () => {
    const ingredients = tracer.trace('get_ingredients', { resource: 'resource_name' }, () => {
      return getIngredients()
    })

    return tracer.trace('assemble_sandwich', { resource: 'resource_name' }, () => {
      assembleSandwich(ingredients)
    })
  })

  res.end(sandwich)
})

To learn more, read API details for tracer.trace().

Promises

Promises can be traced with tracer.trace() which automatically finishes the span when the returned promise resolves, and captures any rejection error automatically.

const getIngredients = () => {
    return new Promise((resolve, reject) => {
        resolve('Salami');
    });
};

app.get('/make-sandwich', (req, res) => {
  return tracer.trace('sandwich.make', { resource: 'resource_name' }, () => {
    return tracer.trace('get_ingredients', { resource: 'resource_name' }, () => getIngredients())
      .then((ingredients) => {
        return tracer.trace('assemble_sandwich', { resource: 'resource_name' }, () => {
          return assembleSandwich(ingredients)
        })
      })
  }).then(sandwich => res.end(sandwich))
})

Async/await

Async/await can be traced with tracer.trace() which automatically finishes the span when the returned promise resolves, and captures any rejection error automatically.

app.get('/make-sandwich', async (req, res) => {
  const sandwich = await tracer.trace('sandwich.make', { resource: 'resource_name' }, async () => {
    const ingredients = await tracer.trace('get_ingredients', { resource: 'resource_name' }, () => {
      return getIngredients()
    })

    return tracer.trace('assemble_sandwich', { resource: 'resource_name' }, () => {
      return assembleSandwich(ingredients)
    })
  })

  res.end(sandwich)
})

Wrapper

You can wrap an existing function without changing its code. This is useful to trace functions for which you don't control the code. This can be done with tracer.wrap() which takes the same arguments as tracer.trace() except its last argument which is the function to wrap instead of a callback.

// After the functions are defined
getIngredients = tracer.wrap('get_ingredients', { resource: 'resource_name' }, getIngredients)
assembleSandwich = tracer.wrap('assemble_sandwich', { resource: 'resource_name' }, assembleSandwich)

// Where routes are defined
app.get('/make-sandwich', (req, res) => {

  const sandwich = tracer.trace('sandwich.make', { resource: 'resource_name' }, () => {
    const ingredients = getIngredients()

    return assembleSandwich(ingredients)
  })

  res.end(sandwich)
})

To learn more, read API details for tracer.wrap().

Request filtering

You may not want some requests of an application to be instrumented. A common case would be health checks or other synthetic traffic. These can be ignored by using the blocklist or allowlist option on the http plugin.

// at the top of the entry point right after tracer.init()
tracer.use('http', {
  blocklist: ['/health', '/ping']
})

This configuration can be split between client and server if needed. For example,

tracer.use('http', {
  server: {
    blocklist: ['/ping']
  }
})

Additionally, traces can be excluded based on their resource name, so that the Agent doesn't send them to Datadog. This and other security and fine-tuning Agent configurations can be found on the Security page or in Ignoring Unwanted Resources.

dd-trace-api

The dd-trace-api package provides a stable public API for Datadog APM's custom Node.js instrumentation. This package implements only the API interface, not the underlying functionality that creates and sends spans to Datadog.

This separation between interface (dd-trace-api) and implementation (dd-trace) offers several benefits:

• You can rely on an API that changes less frequently and more predictably for your custom instrumentation
• If you only use automatic instrumentation, you can ignore API changes entirely
• If you implement both single-step and custom instrumentation, you avoid depending on multiple copies of the dd-trace package

To use dd-trace-api:

1. Install the dd-trace and dd-trace-api libraries in your app. Note: dd-trace is installed for you with single-step instrumentation, but you need to install dd-trace-api manually in your app.

   Copy

   npm install dd-trace dd-trace-api
   

2. Instrument your Node.js application using dd-trace. If you're using single-step instrumentation, you can skip this step.

   Copy

    node --require dd-trace/init app.js
   

3. After this is set up, you can write custom instrumentation exactly like the examples in the previous sections, but you require dd-trace-api instead of dd-trace.

   For example:

const tracer = require('dd-trace-api')
const express = require('express')
const app = express()

app.get('/make-sandwich', (req, res) => {
  const sandwich = tracer.trace('sandwich.make', { resource: 'resource_name' }, () => {
    const ingredients = tracer.trace('get_ingredients', { resource: 'resource_name' }, () => {
      return getIngredients()
    })

    return tracer.trace('assemble_sandwich', { resource: 'resource_name' }, () => {
      assembleSandwich(ingredients)
    })
  })

  res.end(sandwich)
})

See that package's API definition for the full list of supported API calls.

This page details common use cases for adding and customizing observability with Datadog APM.

Adding tags

Add custom span tags to your spans to customize your observability within Datadog. The span tags are applied to your incoming traces, allowing you to correlate observed behavior with code-level information such as merchant tier, checkout amount, or user ID.

Add custom span tags

Add tags directly to a Span interface by calling SetTag:

package main

import (
    "log"
    "net/http"

    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Set tag
    span.SetTag("http.url", r.URL.Path)
    span.SetTag("<TAG_KEY>", "<TAG_VALUE>")
}

func main() {
    tracer.Start(tracer.WithService("<SERVICE_NAME>"))
    defer tracer.Stop()
    http.HandleFunc("/posts", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Datadog's integrations make use of the Context type to propagate the current active span. If you want to add span tags attached to a Context, call the SpanFromContext function:

package main

import (
    "net/http"

    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Retrieve a span for a web request attached to a Go Context.
    if span, ok := tracer.SpanFromContext(r.Context()); ok {
        // Set tag
        span.SetTag("http.url", r.URL.Path)
    }
}

Adding tags globally to all spans

Add tags to all spans by configuring the tracer with the WithGlobalTag option:

package main

import (
    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func main() {
    tracer.Start(
        tracer.WithGlobalTag("datacenter", "us-1"),
        tracer.WithGlobalTag("env", "dev"),
    )
    defer tracer.Stop()
}

Set errors on a span

To set an error on one of your spans, use tracer.WithError as below:

err := someOperation()
span.Finish(tracer.WithError(err))

Adding spans

If you aren't using supported library instrumentation (see Library compatibility), you may want to manually instrument your code.

Manually creating a span

To manually create spans, use the tracer package (see the v2 API on Datadog's godoc; for v1, see the v1 godoc).

You can create spans in two ways:

• Start a child from an existing span with StartChild for v2 or StartSpan for v1.
• Start a span from a context with StartSpanFromContext (see API details for v2 or v1).

//v2: Create a span with a resource name, which is the child of parentSpan.
span := parentSpan.StartChild("mainOp", tracer.ResourceName("/user"))

//v1: Create a span with a resource name, which is the child of parentSpan.
span := tracer.StartSpan("mainOp", tracer.ResourceName("/user"), tracer.ChildOf(parentSpan))

// v1 and v2: Create a span which will be the child of the span in the Context ctx, if there is a span in the context.
// Returns the new span, and a new context containing the new span.
span, ctx := tracer.StartSpanFromContext(ctx, "mainOp", tracer.ResourceName("/user"))

Asynchronous traces

func main() {
	span, ctx := tracer.StartSpanFromContext(context.Background(), "mainOp")
	defer span.Finish()

	go func() {
		asyncSpan := tracer.StartSpanFromContext(ctx, "asyncOp")
		defer asyncSpan.Finish()
		performOp()
	}()
}

Distributed tracing

Create a distributed trace by manually propagating the tracing context:

package main

import (
    "net/http"

    "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    span, ctx := tracer.StartSpanFromContext(r.Context(), "post.process")
    defer span.Finish()

    req, err := http.NewRequest("GET", "http://example.com", nil)
    req = req.WithContext(ctx)
    // Inject the span Context in the Request headers
    err = tracer.Inject(span.Context(), tracer.HTTPHeadersCarrier(req.Header))
    if err != nil {
        // Handle or log injection error
    }
    http.DefaultClient.Do(req)
}