The profiler is shipped within Datadog tracing libraries. If you are already using APM to collect traces for your application, you can skip installing the library and go directly to enabling the profiler.

Requirements

For a summary of the minimum and recommended runtime and tracer versions across all languages, read Supported Language and Tracer Versions.

The Datadog Profiler requires Go 1.19+.

For Code Hotspots and Endpoint Profiling, use dd-trace-go version 1.37.0+.

Continuous Profiler is not supported on serverless platforms, such as AWS Lambda.

Installation

To begin profiling applications:

  1. Ensure Datadog Agent v6+ is installed and running. Datadog recommends using Datadog Agent v7+.

  2. Get dd-trace-go using the command:

    go get gopkg.in/DataDog/dd-trace-go.v1/profiler
    

    Note: Profiler is available in the dd-trace-go library for versions 1.23.0+.

  3. Import the profiler at the start of your application:

    import "gopkg.in/DataDog/dd-trace-go.v1/profiler"
    
  4. Add the following snippet to start the profiler:

    err := profiler.Start(
        profiler.WithService("<SERVICE_NAME>"),
        profiler.WithEnv("<ENVIRONMENT>"),
        profiler.WithVersion("<APPLICATION_VERSION>"),
        profiler.WithTags("<KEY1>:<VALUE1>", "<KEY2>:<VALUE2>"),
        profiler.WithProfileTypes(
          profiler.CPUProfile,
          profiler.HeapProfile,
          // The profiles below are disabled by default to keep overhead
          // low, but can be enabled as needed.
    
          // profiler.BlockProfile,
          // profiler.MutexProfile,
          // profiler.GoroutineProfile,
        ),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer profiler.Stop()
    
  5. Optional: Enable the timeline feature (beta), see prerequisites.

  6. After a minute or two, visualize your profiles in the Datadog APM > Profiler page.

Note: By default, only the CPU and Heap profiles are enabled. Use profiler.WithProfileTypes to enable additional profile types.

Configuration

You can set profiler parameters in code with these functions:

FunctionTypeDescription
WithServiceStringThe Datadog service name, for example, my-web-app.
WithEnvStringThe Datadog environment name, for example, production.
WithVersionStringThe version of your application.
WithTagsList of stringsA list of tags to apply to an uploaded profile. Tags must be of the format <KEY>:<VALUE>.

Alternatively you can set profiler configuration using environment variables:

Environment variableTypeDescription
DD_ENVStringThe environment name, for example, production.
DD_SERVICEStringThe service name, for example, web-backend.
DD_VERSIONStringThe version of your service.
DD_TAGSStringTags to apply to an uploaded profile. Must be a list of <key>:<value> separated by commas such as: layer:api,team:intake.

Showing C function calls in CPU profiles

By default, Go’s CPU profiler only shows detailed information for Go code. If your program calls C code, the time spent running C code is reflected in the profile, but the call stacks only show Go function calls.

To add detailed C function call information to CPU profiles, you may opt to use library such as ianlancetaylor/cgosymbolizer. To use this library:

  1. Download the package:

    go get github.com/ianlancetaylor/cgosymbolizer@latest
    
  2. Add the following import anywhere in your program:

    import _ "github.com/ianlancetaylor/cgosymbolizer"
    

Note: This library is considered experimental. It can cause (infrequent) deadlocks in programs that use C++ exceptions, or that use libraries such as tcmalloc, which also collect call stacks.

Save 2 to 14% CPU in production with PGO

Starting Go 1.21, the Go compiler supports Profile-Guided Optimization (PGO). PGO enables additional optimizations on code identified as hot by CPU profiles of production workloads. This is compatible with Datadog Go Continuous Profiler and can be used for production builds.

Follow this guide to set it up.

Not sure what to do next?

The Getting Started with Profiler guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem.

Further Reading