Go Runtime Metrics

Automatic configuration

To enable Go runtime metrics collection, start the tracer using the WithRuntimeMetrics option:

tracer.Start(tracer.WithRuntimeMetrics())

View runtime metrics in correlation with your Go services on the Service Catalog in Datadog.

By default, runtime metrics from your application are sent every 10 seconds to the Datadog Agent with DogStatsD. Make sure that DogStatsD is enabled for the Agent. If your Datadog Agent DogStatsD address differs from the default localhost:8125, use the WithDogstatsdAddress option (available starting in 1.18.0) or the environment variables DD_AGENT_HOST and DD_DOGSTATSD_PORT.

If WithDogstatsdAddress is not used, the tracer attempts to determine the address of the statsd service according to the following rules:

  1. Look for /var/run/datadog/dsd.socket and use it if present. IF NOT, continue to #2.
  2. The host is determined by DD_AGENT_HOST, and defaults to “localhost”.
  3. The port is retrieved from the Agent. If not present, it is determined by DD_DOGSTATSD_PORT, and defaults to 8125.

If you are running the Agent as a container, ensure that DD_DOGSTATSD_NON_LOCAL_TRAFFIC is set to true, and that port 8125 is open on the Agent. Additionally, for Kubernetes or ECS, follow the guidelines below:

Alternatively, the Agent can ingest metrics with a Unix Domain Socket (UDS) as an alternative to UDP transport. For more information, read DogStatsD over Unix Domain Socket.

Data Collected

The following metrics are collected by default after enabling Go metrics.

runtime.go.num_cpu
(gauge)		CPUs detected by the runtime.
Shown as resource
runtime.go.num_goroutine
(gauge)		goroutines spawned.
Shown as invocation
runtime.go.num_cgo_call
(gauge)		CGO calls made.
Shown as invocation
runtime.go.mem_stats.alloc
(gauge)		Alloc is bytes of allocated heap objects.
Shown as byte
runtime.go.mem_stats.total_alloc
(gauge)		TotalAlloc is cumulative bytes allocated for heap objects.
Shown as byte
runtime.go.mem_stats.sys
(gauge)		Sys is the total bytes of memory obtained from the OS.
Shown as byte
runtime.go.mem_stats.lookups
(gauge)		Lookups is the number of pointer lookups performed by the
Shown as unit
runtime.go.mem_stats.mallocs
(gauge)		Mallocs is the cumulative count of heap objects allocated.
Shown as unit
runtime.go.mem_stats.frees
(gauge)		Frees is the cumulative count of heap objects freed.
Shown as unit
runtime.go.mem_stats.heap_alloc
(gauge)		HeapAlloc is bytes of allocated heap objects.
Shown as byte
runtime.go.mem_stats.heap_sys
(gauge)		HeapSys is bytes of heap memory obtained from the OS.
Shown as byte
runtime.go.mem_stats.heap_idle
(gauge)		HeapIdle is bytes in idle (unused) spans.
Shown as byte
runtime.go.mem_stats.heap_inuse
(gauge)		HeapInuse is bytes in in-use spans.
Shown as byte
runtime.go.mem_stats.heap_released
(gauge)		HeapReleased is bytes of physical memory returned to the OS.
Shown as byte
runtime.go.mem_stats.heap_objects
(gauge)		HeapObjects is the number of allocated heap objects.
Shown as unit
runtime.go.mem_stats.stack_inuse
(gauge)		StackInuse is bytes in stack spans.
Shown as byte
runtime.go.mem_stats.stack_sys
(gauge)		StackSys is bytes of stack memory obtained from the OS.
Shown as byte
runtime.go.mem_stats.m_span_inuse
(gauge)		MSpanInuse is bytes of allocated mspan structures.
Shown as byte
runtime.go.mem_stats.m_span_sys
(gauge)		MSpanSys is bytes of memory obtained from the OS for mspan structures.
Shown as byte
runtime.go.mem_stats.m_cache_inuse
(gauge)		MCacheInuse is bytes of allocated mcache structures.
Shown as byte
runtime.go.mem_stats.m_cache_sys
(gauge)		MCacheSys is bytes of memory obtained from the OS for
Shown as byte
runtime.go.mem_stats.buck_hash_sys
(gauge)		BuckHashSys is bytes of memory in profiling bucket hash tables.
Shown as byte
runtime.go.mem_stats.gc_sys
(gauge)		GCSys is bytes of memory in garbage collection metadata.
Shown as byte
runtime.go.mem_stats.other_sys
(gauge)		OtherSys is bytes of memory in miscellaneous off-heap
Shown as byte
runtime.go.mem_stats.next_gc
(gauge)		NextGC is the target heap size of the next GC cycle.
Shown as byte
runtime.go.mem_stats.last_gc
(gauge)		LastGC is the time the last garbage collection finished, as nanoseconds since 1970 (the UNIX epoch).
Shown as nanosecond
runtime.go.mem_stats.pause_total_ns
(gauge)		PauseTotalNs is the cumulative nanoseconds in GC
Shown as nanosecond
runtime.go.mem_stats.num_gc
(gauge)		NumGC is the number of completed GC cycles.
Shown as unit
runtime.go.mem_stats.num_forced_gc
(gauge)		NumForcedGC is the number of GC cycles that were forced by the application calling the GC function.
Shown as unit
runtime.go.mem_stats.gc_cpu_fraction
(gauge)		GCCPUFraction is the fraction of this program's available CPU time used by the GC since the program started.
Shown as fraction
runtime.go.gc_stats.pause_quantiles.min
(gauge)		Distribution of GC pause times: minimum values
Shown as nanosecond
runtime.go.gc_stats.pause_quantiles.25p
(gauge)		Distribution of GC pause times: 25th percentile
Shown as nanosecond
runtime.go.gc_stats.pause_quantiles.75p
(gauge)		Distribution of GC pause times: 50th percentile
Shown as nanosecond
runtime.go.gc_stats.pause_quantiles.95p
(gauge)		Distribution of GC pause times: 75th percentile
Shown as nanosecond
runtime.go.gc_stats.pause_quantiles.max
(gauge)		Distribution of GC pause times: maximum values
Shown as nanosecond

Along with displaying these metrics in your APM Service Page, Datadog provides a default Go Runtime Dashboard.

Further Reading

Additional helpful documentation, links, and articles:

Connect your Logs and Traces togetherDOCUMENTATION more more Manually instrument your application to create traces.DOCUMENTATION more more Explore your services, resources, and tracesDOCUMENTATION more more