Search Profiles

Each row is a profile of a process for a short amount of time. By default, profiles are uploaded once a minute. Depending on the language, these processes are profiled between 15s and 60s.

You can filter according to infrastructure tags or application tags set up from your environment tracing configuration. By default the following facets are available:

FacetDefinition
EnvThe environment your application is running on (production, staging).
ServiceThe name of the service your code is running.
VersionThe version of your code.
HostThe hostname your profiled process is running on.
RuntimeThe type of runtime the profiled process is running (JVM, CPython).

The following measures are available:

MeasureDefinition
CPUCPU usage, measured in cores.
Memory AllocationMemory allocation rate over the course of the profile. This value can be above the amount of memory on your system because allocated memory can be garbage collected during the profile.

Profiles

Click on a line to view a specific profile:

The header contains information associated with your profile, like the service that generated it, or the environment and code version associated to it.

Four tabs are below the profile header:

TabDefinition
ProfilesA flame graph and summary table of the profile you are looking at. You can switch between profile types (CPU, Memory allocation).
AnalysisA set of heuristics that suggest potential issues or areas of improvement in your code. Currently only available for Java.
MetricsProfiler metrics coming from all profiles of the same service.
Runtime InfoRuntime properties in supported languages, and profile tags.

Note: In the upper right corner of each profile, there are options to:

  • Download the profile
  • Switch the profile to full screen

Profile types

In the Profiles tab, you can see all profile types available for a given language. Depending on the language, the information collected about your profile differs.

Once enabled, the following profile types are collected:

CPU
Shows the time each method spent running on the CPU. It includes your code that runs in the JVM (Java, Kotlin, etc), but not JVM operations or native code called from within the JVM.
Allocations
Shows the amount of heap memory allocated by each method, including allocations which were subsequently freed.
Wall Time in Native Code
Shows the elapsed time spent in native code. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the method is running. This profile does not include time spent running JVM bytecode, which is typically most of your application code.
Class Load
Shows the number of classes loaded by each method.
Thrown Exceptions
Shows the number of errors and exceptions thrown by each method, as well as their type.
File I/O
Shows the time each method spent reading from and writing to files.
Lock
Shows the time each method spent waiting for a lock.
Socket I/O
Shows the time each method spent reading from and writing to socket I/O.

Once enabled, the following profile types are collected:

CPU
Shows the time each function spent running on the CPU, including Python and native code.
Wall Time
Shows the elapsed time used by each function. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the function is running.
Heap Live Size
Shows the amount of heap memory allocated by each function that has not been garbage collected (yet). This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.
Allocated Memory
Shows the amount of heap memory allocated by each function, including allocations which were subsequently freed - only supported with Python 3.
Allocations
Shows the number of heap allocations made by each function, including allocations which were subsequently freed.
Thrown Exceptions
Shows the number of caught or uncaught exceptions raised by each function, as well as their type.
Lock Wait Time
Shows the time each function spent waiting for a lock.
Locked Time
Shows the time each function spent holding a lock.
Lock Acquires
Shows the number of times each function acquired a lock.
Lock Releases
Shows the number of times each function released a lock.

Once enabled, the following profile types are collected:

CPU Time
Shows the time each function spent running on the CPU. Off-CPU time such as waiting for Networking, Channels, Mutexes and Sleep are not captured in this profile. See Mutex and Block profiles.
Allocations
Shows the number of objects allocated by each function in heap memory during the profiling period (default: 60s), including allocations which were subsequently freed. Go calls this alloc_objects. Stack allocations are not tracked. This is useful for investigating garbage collection load. See also the note about how this measure changes in version 1.33.0 in Delta profiles.
Allocated Memory
Shows the amount of heap memory allocated by each function during the profiling period (default: 60s), including allocations which were subsequently freed. Go calls this alloc_space. Stack allocations are not tracked. This is useful for investigating garbage collection load. See also the note about how this measure changes in version 1.33.0 in Delta profiles.
Heap Live Objects
Shows the number of objects allocated by each function in heap memory that have not been garbage collected (yet). Go calls this inuse_objects. This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.
Heap Live Size
Shows the amount of heap memory allocated by each function that has not been garbage collected (yet). Go calls this inuse_space. This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.
Mutex
Shows the time functions have been waiting on mutexes during the profiling period (default: 60s). The stack traces in this profile point the Unlock() operation that allowed another goroutine blocked on the mutex to proceed. Short mutex contentions using spinlocks are not captured by this profile, but can be seen in the CPU profile. See also the note about how this measure changes in version 1.33.0 in Delta profiles.
Block
Shows the time functions have been waiting on mutexes and channel operations during the profiling period (default: 60s). Sleep, GC, Network and Syscall operations are not captured by this profile. Blocking operations are only captured after they become unblocked, so this profile cannot be used to debug applications that appear to be stuck. For mutex contentions, the stack traces in this profile point to blocked Lock() operations. This will tell you where your program is getting blocked, while the mutex profile tells you what part of your program is causing the contention. See our Block Profiling in Go research for more in-depth information. See also the note about how this measure changes in version 1.33.0 in Delta profiles. Note: The block profiler can cause noticeable overhead for production workloads. If enabling it in production, prefer high rates (such as 100000000, that is 100 milliseconds) and look for signs of increased latency or CPU utilization.
Goroutines
Shows a snapshot of the number of goroutines currently executing the same functions (On CPU as well as waiting Off-CPU). An increasing number of goroutines between snapshots can indicate that the program is leaking goroutines. In most healthy applications this profile is dominated by worker pools and shows the number of goroutines they use. Applications that are extremely latency sensitive and use a large number of goroutines (> 10.000) should be aware that enabling this profile requires stop-the-world pauses. The pauses occur only once every profiling period (default 60s) and normally last for ~1µsec per goroutine. Typical applications with a p99 latency SLO of ~100ms can generally ignore this warning. See our Goroutine Profiling in Go research for more in-depth information.

Delta profiles

Note: In Go profiler versions before 1.33.0, Allocations, Allocated Memory, Mutex, and Block metrics are shown as measures accumulated since the process was started, as opposed to during the profiling period. The change to delta profiles in version 1.33.0 lets you see how these measures are changing instead of accumulating. Delta profiling is on by default. Profiler version 1.35.0 allows you to disable delta profiles using the WithDeltaProfiles option.

As of profiler version 1.37.0, accumulated profiles are no longer uploaded when delta profiling is enabled to reduce upload bandwidth usage. Contact Support to discuss your use case if you rely on the full accumulated profiles.

Once enabled, the following profile types are collected:

CPU
Shows the time each function spent running on the CPU, including Ruby and native code.
Wall Time
Shows the elapsed time used by each function. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the function is running.

Once enabled, the following profile types are collected:

Wall Time
Shows the elapsed time used by each function. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the function is running.
Heap Live Size
Shows the amount of heap memory allocated by each function that has not been garbage collected (yet). This is useful for investigating the overall memory usage of your service and identifying potential memory leaks.

Once enabled, the following profile types are collected:

Wall Time
Shows the elapsed time spent in managed methods. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the method is running.
CPU
Shows the time each method spent running on the CPU.
Thrown Exceptions
Shows the number of caught or uncaught exceptions raised by each method, as well as their type and message.

Once enabled, the following profile types are collected:

Wall Time
Shows the elapsed time used by each function. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the function is running.

Once enabled, the following profile types are collected:

CPU
Shows the time each function spent running on the CPU.

Further Reading