Tracing Go Applications

Compatibility requirements

The Go Tracer requires Go 1.16+ and Datadog Agent >= 5.21.1. For a full list of supported libraries, see the Compatibility Requirements page.

Installation and getting started

For configuration instructions and details about using the API, see the Datadog API documentation.

For a description of the terminology used in APM, see the Getting started with APM section. For details about contributing, check the official repository README.md.

Use the migration document if you need to migrate from an older version of the tracer (for example, v<0.6.x) to the newest version.

When you set up tracing, you’re also setting up Continuous Profiler, and you need only enable Profiler to start receiving profiling data from your app.

Installation

Follow the in-app documentation (recommended)

Follow the Quickstart instructions within the Datadog for the best experience, including:

Step-by-step instructions scoped to your deployment configuration (hosts, Docker, Kubernetes, or Amazon ECS).
Dynamically set service, env, and version tags.
Enable the Continuous Profiler, ingesting 100% of traces , and Trace ID injection into logs during setup.

Otherwise, follow the instructions below to add the Datadog Tracing Library to your code.

Automatic instrumentation

Datadog has a series of pluggable packages which provide out-of-the-box support for instrumenting a series of libraries and frameworks. A list of these packages can be found in the Compatibility Requirements page. To trace these integrations, import these packages into your application and follow the configuration instructions listed alongside each Integration.

Configuration

Datadog recommends using DD_ENV, DD_SERVICE, and DD_VERSION to set env, service, and version for your services.

Read the Unified Service Tagging documentation for recommendations on how to configure these environment variables. These variables are available for versions 1.24.0+ of the Go tracer.

You may also elect to provide env, service, and version through the tracer’s API:

package main

import (
    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func main() {
    tracer.Start(
        tracer.WithEnv("prod"),
        tracer.WithService("test-go"),
        tracer.WithServiceVersion("abc123"),
    )

    // When the tracer is stopped, it will flush everything it has to the Datadog Agent before quitting.
    // Make sure this line stays in your main function.
    defer tracer.Stop()
}

The Go tracer supports additional environment variables and functions for configuration. See all available options in the configuration documentation.

DD_VERSION: Set the application’s version, for example: 1.2.3, 6c44da20, 2020.02.13
DD_SERVICE: The service name to be used for this application.
DD_ENV: Set the application’s environment, for example: prod, pre-prod, staging.
DD_AGENT_HOST: Default: localhost
Override the default trace Agent host address for trace submission.
DD_DOGSTATSD_PORT: Default: 8125
Override the default trace Agent port for DogStatsD metric submission.
DD_TRACE_SAMPLE_RATE: Enable ingestion rate control.
DD_TRACE_RATE_LIMIT: Maximum number of spans to sample per-second, per-Go process. Defaults to 100 when DD_TRACE_SAMPLE_RATE is set. Otherwise, delegates rate limiting to the Datadog Agent.
DD_TAGS: Default: []
A list of default tags to be added to every span and profile. Tags can be separated by commas or spaces, for example: layer:api,team:intake or layer:api team:intake
DD_TRACE_STARTUP_LOGS: Default: true
Enable startup configuration and the diagnostic log.
DD_TRACE_DEBUG: Default: false
Enable debug logging in the tracer.
DD_TRACE_ENABLED: Default: true
Enable web framework and library instrumentation. When false, the application code doesn’t generate any traces.
DD_SERVICE_MAPPING: Default: null
Dynamically rename services through configuration. Services can be separated by commas or spaces, for example: mysql:mysql-service-name,postgres:postgres-service-name, mysql:mysql-service-name postgres:postgres-service-name.

Configure the Datadog Agent for APM

Install and configure the Datadog Agent to receive traces from your now instrumented application. By default the Datadog Agent is enabled in your datadog.yaml file under apm_config with enabled: true and listens for trace traffic at localhost:8126. For containerized environments, follow the links below to enable trace collection within the Datadog Agent.

Set apm_non_local_traffic: true in the apm_config section of your main datadog.yaml configuration file.
See the specific setup instructions to ensure that the Agent is configured to receive traces in a containerized environment:

After the application is instrumented, the trace client attempts to send traces to the Unix domain socket /var/run/datadog/apm.socket by default. If the socket does not exist, traces are sent to http://localhost:8126.

A similar rule applies to all metrics sent by the Go tracer (including Runtime Metrics and internal telemetry): the client attempts to send Dogstatsd data to the Unix domain socket /var/run/datadog/dsd.socket and defaults to http://localhost:8125 if that does not exist.

If you require different hosts or ports, use one or more of the following environment variables. The examples show the defaults, but you can set them to other values as well.

DD_AGENT_HOST=localhost   # The host to send traces and metrics to. Defaults to localhost.
DD_TRACE_AGENT_PORT=8126  # The port to send traces to. Defaults to 8126.
DD_DOGSTATSD_PORT=8125    # The port to send Dogstatsd metrics to. Defaults to 8125.

The connection for traces can also be configured in code:

package main

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start(
        // Unix Domain Socket configuration:
        tracer.WithUDS("/var/run/datadog/apm.socket"),
        // or, for a non-default TCP connection:
        // tracer.WithAgentAddr("localhost:8126"),
        // or, for an alternative UDP connection for Dogstatsd:
        // tracer.WithDogstatsdAddress("localhost:8125"),
    )
    defer tracer.Stop()

    // ...
}

To set up Datadog APM in AWS Lambda, see the Tracing Serverless Functions documentation.

Tracing is available for a number of other environments, such as Heroku, Cloud Foundry, AWS Elastic Beanstalk, and Azure App Service.

For other environments, please refer to the Integrations documentation for that environment and contact support if you are encountering any setup issues.

Configure APM environment name

The APM environment name may be configured in the agent or using the WithEnv start option of the tracer.

B3 headers extraction and injection

The Datadog APM tracer supports B3 headers extraction and injection for distributed tracing.

Distributed headers injection and extraction is controlled by configuring injection/extraction styles. Two styles are supported: Datadog and B3.

Configure injection styles using the environment variable DD_PROPAGATION_STYLE_INJECT=Datadog,B3

Configure extraction styles using the environment variable DD_PROPAGATION_STYLE_EXTRACT=Datadog,B3

The values of these environment variables are comma separated lists of header styles that are enabled for injection or extraction. By default only the Datadog extraction style is enabled.

If multiple extraction styles are enabled, extraction attempts are made in the order that those styles are specified. The first successfully extracted value is used.