The profiler is shipped within Datadog tracing libraries. If you are already using APM to collect traces for your application, you can skip installing the library and go directly to enabling the profiler.

Requirements

The Datadog Profiler requires Go 1.12+.

For Code Hotspots and Endpoint Profiling, use Go version 1.18+ and dd-trace-go version 1.37.0+.

Continuous Profiler is not supported on serverless platforms, such as AWS Lambda.

Installation

To begin profiling applications:

  1. If you are already using Datadog, upgrade your agent to version 7.20.2+ or 6.20.2+.

  2. Get dd-trace-go using the command:

    go get gopkg.in/DataDog/dd-trace-go.v1/profiler
    

    Note: Profiler is available in the dd-trace-go library for versions 1.23.0+.

  3. Import the profiler at the start of your application:

    import "gopkg.in/DataDog/dd-trace-go.v1/profiler"
    
  4. Add the following snippet to start the profiler:

    err := profiler.Start(
        profiler.WithService("<SERVICE_NAME>"),
        profiler.WithEnv("<ENVIRONMENT>"),
        profiler.WithVersion("<APPLICATION_VERSION>"),
        profiler.WithTags("<KEY1>:<VALUE1>", "<KEY2>:<VALUE2>"),
        profiler.WithProfileTypes(
          profiler.CPUProfile,
          profiler.HeapProfile,
          // The profiles below are disabled by default to keep overhead
          // low, but can be enabled as needed.
    
          // profiler.BlockProfile,
          // profiler.MutexProfile,
          // profiler.GoroutineProfile,
        ),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer profiler.Stop()
    
  5. Optional: Enable the timeline feature (beta), see prerequisites.

  6. After a minute or two, visualize your profiles in the Datadog APM > Profiler page.

Note: By default, only the CPU and Heap profiles are enabled. Use profiler.WithProfileTypes to enable additional profile types.

Configuration

You can set profiler parameters in code with these functions:

FunctionTypeDescription
WithServiceStringThe Datadog service name, for example, my-web-app.
WithEnvStringThe Datadog environment name, for example, production.
WithVersionStringThe version of your application.
WithTagsList of stringsA list of tags to apply to an uploaded profile. Tags must be of the format <KEY>:<VALUE>.

Alternatively you can set profiler configuration using environment variables:

Environment variableTypeDescription
DD_ENVStringThe environment name, for example, production.
DD_SERVICEStringThe service name, for example, web-backend.
DD_VERSIONStringThe version of your service.
DD_TAGSStringTags to apply to an uploaded profile. Must be a list of <key>:<value> separated by commas such as: layer:api,team:intake.

Showing C function calls in CPU profiles

By default, Go’s CPU profiler only shows detailed information for Go code. If your program calls C code, the time spent running C code is reflected in the profile, but the call stacks only show Go function calls.

To add detailed C function call information to CPU profiles, you may opt to use library such as ianlancetaylor/cgosymbolizer. To use this library:

  1. Download the package:

    go get github.com/ianlancetaylor/cgosymbolizer@latest
    
  2. Add the following import anywhere in your program:

    import _ "github.com/ianlancetaylor/cgosymbolizer"
    

Note: This library is considered experimental. It can cause (infrequent) deadlocks in programs that use C++ exceptions, or that use libraries such as tcmalloc, which also collect call stacks.

Not sure what to do next?

The Getting Started with Profiler guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem.

Further Reading