Go Custom Instrumentation

If you have not yet read the instructions for auto-instrumentation and setup, start with the Go Setup Instructions.

This page details common use cases for adding and customizing observability with Datadog APM.

Adding tags

Add custom span tags to your spans to customize your observability within Datadog. The span tags are applied to your incoming traces, allowing you to correlate observed behavior with code-level information such as merchant tier, checkout amount, or user ID.

Add custom span tags

Add tags directly to a Span interface by calling SetTag:

package main

import (
    "log"
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Create a span for a web request at the /posts URL.
    span := tracer.StartSpan("web.request", tracer.ResourceName("/posts"))
    defer span.Finish()

    // Set tag
    span.SetTag("http.url", r.URL.Path)
    span.SetTag("<TAG_KEY>", "<TAG_VALUE>")
}

func main() {
    tracer.Start(tracer.WithService("<SERVICE_NAME>"))
    defer tracer.Stop()
    http.HandleFunc("/posts", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Datadog’s integrations make use of the Context type to propagate the current active span. If you want to add span tags attached to a Context, call the SpanFromContext function:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Retrieve a span for a web request attached to a Go Context.
    if span, ok := tracer.SpanFromContext(r.Context()); ok {
        // Set tag
        span.SetTag("http.url", r.URL.Path)
    }
}

Adding tags globally to all spans

Add tags to all spans by configuring the tracer with the WithGlobalTag option:

package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start(
        tracer.WithGlobalTag("datacenter", "us-1"),
        tracer.WithGlobalTag("env", "dev"),
    )
    defer tracer.Stop()
}

Set errors on a span

To set an error on one of your spans, use tracer.WithError as below:

err := someOperation()
span.Finish(tracer.WithError(err))

Adding spans

If you aren’t using supported library instrumentation (see Library compatibility), you may want to to manually instrument your code.

Manually creating a new span

To make use of manual instrumentation, use the tracer package which is documented on Datadog’s godoc page:

There are two functions available to create spans. API details are available for StartSpan here and for StartSpanFromContext here.

//Create a span with a resource name, which is the child of parentSpan.
span := tracer.StartSpan("mainOp", tracer.ResourceName("/user"), tracer.ChildOf(parentSpan))

// Create a span which will be the child of the span in the Context ctx, if there is a span in the context.
// Returns the new span, and a new context containing the new span.
span, ctx := tracer.StartSpanFromContext(ctx, "mainOp", tracer.ResourceName("/user"))

Asynchronous traces

func main() {
	span, ctx := tracer.StartSpanFromContext(context.Background(), "mainOp")
	defer span.Finish()

	go func() {
		asyncSpan := tracer.StartSpanFromContext(ctx, "asyncOp")
		defer asyncSpan.Finish()
		performOp()
	}()
}

Distributed tracing

Create a distributed trace by manually propagating the tracing context:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    span, ctx := tracer.StartSpanFromContext(r.Context(), "post.process")
    defer span.Finish()

    req, err := http.NewRequest("GET", "http://example.com", nil)
    req = req.WithContext(ctx)
    // Inject the span Context in the Request headers
    err = tracer.Inject(span.Context(), tracer.HTTPHeadersCarrier(req.Header))
    if err != nil {
        // Handle or log injection error
    }
    http.DefaultClient.Do(req)
}

Then, on the server side, to continue the trace, start a new Span from the extracted Context:

package main

import (
    "net/http"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    // Extract the span Context and continue the trace in this service
    sctx, err := tracer.Extract(tracer.HTTPHeadersCarrier(r.Header))
    if err != nil {
        // Handle or log extraction error
    }

    span := tracer.StartSpan("post.filter", tracer.ChildOf(sctx))
    defer span.Finish()
}

Trace client and Agent configuration

There are additional configurations possible for both the tracing client and Datadog Agent for context propagation with B3 Headers, as well as excluding specific resources from sending traces to Datadog in the event these traces are not wanted in metrics calculated, such as Health Checks.

Trace context propagation for distributed tracing

The Datadog APM tracer supports extraction and injection of B3 and W3C headers for distributed tracing.

Distributed headers injection and extraction is controlled by configuring injection/extraction styles. Supported styles are: tracecontext, Datadog, B3 and B3 single header.

  • Configure injection styles using the DD_PROPAGATION_STYLE_INJECT=tracecontext,B3 environment variable.
  • Configure extraction styles using the DD_PROPAGATION_STYLE_EXTRACT=tracecontext,B3 environment variable.
  • Configure both injection and extraction styles using the DD_TRACE_PROPAGATION_STYLE=tracecontext,B3 environment variable.

The values of these environment variables are comma-separated lists of header styles enabled for injection or extraction. By default, the tracecontext,Datadog styles are enabled.

To disable trace context propagation, set the value of the environment variables to none.

  • Disable injection styles using the DD_PROPAGATION_STYLE_INJECT=none environment variable.
  • Disable extraction styles using the DD_PROPAGATION_STYLE_EXTRACT=none environment variable.
  • Disable all trace context propagation (both inject and extract) using the DD_PROPAGATION_STYLE=none environment variable.

If multiple environment variables are set, DD_PROPAGATION_STYLE_INJECT and DD_PROPAGATION_STYLE_EXTRACT override any value provided in DD_PROPAGATION_STYLE.

If multiple extraction styles are enabled, extraction attempts are made in the order that those styles are specified. The first successfully extracted value is used.

Resource filtering

Traces can be excluded based on their resource name, to remove synthetic traffic such as health checks from reporting traces to Datadog. This and other security and fine-tuning configurations can be found on the Security page.

Further Reading