Go - Save up to 14% CPU in Production with Profile-Guided Optimization

Overview

Starting with Go 1.21, the Go compiler supports profile-guided optimization (PGO).

PGO enables additional optimizations on code identified as hot by CPU profiles of production workloads. This is compatible with Datadog Go Continuous Profiler and can be used for production builds.

How PGO works

Here are some key points about how PGO works:

  • Building a Go program with PGO enabled causes the compiler to look for a pprof CPU profile named default.pgo and use it to produce a more optimized binary.
  • Typical programs should see a 2-14% decrease in CPU time after optimization. PGO remains under active development, and future versions of Go aim to achieve even greater CPU savings. Datadog is actively supporting this initiative.
  • PGO produces the best results when using representative profiles. However, using non-representative or old profiles (from previous versions of the software) is not expected to result in slower binaries compared to not using PGO.
  • Using a profile from a PGO optimized application is not expected to cause optimization/deoptimization cycles. This is referred to as iterative stability.

For more information, see the Go PGO documentation.

Enabling PGO

PGO is a standard Go compiler option that you can use by manually downloading profiles from Datadog. Datadog built a tool, datadog-pgo to help you enable PGO on all services, using the latest and most representative profiles.

To enable PGO using the datadog-pgo tool:

  1. Create a dedicated API key and an application key scoped to at least continuous_profiler_pgo_read as described in API and Application Keys.

  2. Set DD_API_KEY, DD_APP_KEY, and DD_SITE with the environment secret mechanism of your CI provider.

  3. Run datadog-pgo before your build step. For example, for a service foo that runs in prod and has its main package in ./cmd/foo, you should add this step:

    go run github.com/DataDog/datadog-pgo@latest "service:foo env:prod" ./cmd/foo/default.pgo

The Go toolchain automatically picks up any default.pgo file in the main package, so you don’t need to modify your go build step.

See the datadog-pgo GitHub repository for more details.

Checking if PGO is enabled

To check where PGO is enabled, search for Go profiles without pgo tag set to true.

The pgo tag was implemented in dd-trace-go 1.61.0, so any profiles prior to this version will not have pgo:false.

Further reading

Additional helpful documentation, links, and articles:

Datadog Continuous ProfilerDOCUMENTATION more more Comparing ProfilesDOCUMENTATION more more