Go Custom Instrumentation

If you have not yet read the instructions for auto-instrumentation and setup, start with the Go Setup Instructions.

This page details common use cases for adding and customizing observability with Datadog APM.

Adding Tags

Add custom span tags to your spans to customize your observability within Datadog. The span tags are applied to your incoming traces, allowing you to correlate observed behavior with code-level information such as merchant tier, checkout amount, or user ID.

Add custom span tags

Add tags directly to a Span interface by calling SetTag:

package main

import (
    "log"
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Set tag
    span.SetTag("http.url", r.URL.Path)
    span.SetTag("<TAG_KEY>", "<TAG_VALUE>")
}

func main() {
    tracer.Start(tracer.WithServiceName("<SERVICE_NAME>"))
    defer tracer.Stop()
    http.HandleFunc("/posts", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Datadog’s integrations make use of the Context type to propagate the current active span. If you want to add span tags attached to a Context, call the SpanFromContext function:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Retrieve a span for a web request attached to a Go Context.
    if span, ok := tracer.SpanFromContext(r.Context()); ok {
        // Set tag
        span.SetTag("http.url", r.URL.Path)
    }
}

Adding tags globally to all spans

Add tags to all spans by configuring the tracer with the WithGlobalTag option:

package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start(
        tracer.WithGlobalTag("datacenter", "us-1"),
        tracer.WithGlobalTag("env", "dev"),
    )
    defer tracer.Stop()
}

Set errors on a span

To set an error on one of your spans, use tracer.WithError as below:

err := someOperation()
span.Finish(tracer.WithError(err))

Adding spans

If you aren’t using supported library instrumentation (see Library compatibility), you may want to to manually instrument your code.

Manually creating a new span

To make use of manual instrumentation, use the tracer package which is documented on Datadog’s godoc page:

There are two functions available to create spans. API details are available for StartSpan here and for StartSpanFromContext here.

//Create a span with a resource name, which is the child of parentSpan.
span := tracer.StartSpan(“mainOp”, tracer.ResourceName("/user"), tracer.ChildOf(parentSpan))

// Create a span which will be the child of the span in the Context ctx, if there is a span in the context.
// Returns the new span, and a new context containing the new span.
span, ctx := tracer.StartSpanFromContext(ctx, “mainOp”, tracer.ResourceName("/user"))

Asynchronous Traces

func main() {
	span, ctx := tracer.StartSpanFromContext(context.Background(), “mainOp”)
	defer span.Finish()

	go func() {
		asyncSpan := tracer.StartSpanFromContext(ctx, “asyncOp”)
		defer asyncSpan.Finish()
		performOp()
	}()
}

Distributed Tracing

Create a distributed trace by manually propagating the tracing context:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    span, ctx := tracer.StartSpanFromContext(r.Context(), "post.process")
    defer span.Finish()

    req, err := http.NewRequest("GET", "http://example.com", nil)
    req = req.WithContext(ctx)
    // Inject the span Context in the Request headers
    err = tracer.Inject(span.Context(), tracer.HTTPHeadersCarrier(req.Header))
    if err != nil {
        // Handle or log injection error
    }
    http.DefaultClient.Do(req)
}

Then, on the server side, to continue the trace, start a new Span from the extracted Context:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Extract the span Context and continue the trace in this service
    sctx, err := tracer.Extract(tracer.HTTPHeadersCarrier(r.Header))
    if err != nil {
        // Handle or log extraction error
    }

    span := tracer.StartSpan("post.filter", tracer.ChildOf(sctx))
    defer span.Finish()
}

Trace Client & Agent Configuration

There are additional configurations possible for both the tracing client and Datadog Agent for context propagation with B3 Headers, as well as excluding specific resources from sending traces to Datadog in the event these traces are not wanted in metrics calculated, such as Health Checks.

B3 Headers Extraction and Injection

The Datadog APM tracer supports B3 headers extraction and injection for distributed tracing.

Distributed headers injection and extraction is controlled by configuring injection/extraction styles. Two styles are supported: Datadog and B3.

Configure injection styles using the environment variable: DD_PROPAGATION_STYLE_INJECT=Datadog,B3

Configure extraction styles using the environment variable: DD_PROPAGATION_STYLE_EXTRACT=Datadog,B3

The values of these environment variables are comma separated lists of header styles that are enabled for injection or extraction. By default, the Datadog extraction style is enabled.

If multiple extraction styles are enabled, extraction attempts are made in the order that those styles are specified. The first successfully extracted value is used.

Resource Filtering

Traces can be excluded based on their resource name, to remove synthetic traffic such as health checks from reporting traces to Datadog. This and other security and fine-tuning configurations can be found on the Security page.