Profiling
Rapport de recherche Datadog : Bilan sur l'adoption de l'informatique sans serveur Rapport : Bilan sur l'adoption de l'informatique sans serveur

Profiling

Cette page n'est pas encore disponible en français, sa traduction est en cours.
Si vous avez des questions ou des retours sur notre projet de traduction actuel, n'hésitez pas à nous contacter.
Datadog Profiling is in beta. Reach out to Datadog Support if you encounter any issues or have feedback to share.

Setup

Profiling libraries are shipped within the following tracing language libraries. Select your language below to learn how to enable profiling for your application:

The Datadog Profiler requires Java Flight Recorder. The Datadog Profiling library is supported in OpenJDK 11+, Oracle Java 11+, and Zulu Java 8+ (minor version 1.8.0_212+). All JVM-based languages, such as Scala, Groovy, Kotlin, Clojure, etc. are supported. To begin profiling applications:

  1. Download dd-java-agent.jar, which contains the Java Agent class files, and add the dd-trace-java version to your pom.xml or equivalent:

    wget -O dd-java-agent.jar 'https://repository.sonatype.org/service/local/artifact/maven/redirect?r=central-proxy&g=com.datadoghq&a=dd-java-agent&v=LATEST'

    Note: Profiling is available in the dd-java-agent.jar library in versions 0.44+.

  2. Update your service invocation to look like:

    java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.api-key-file=<API_KEY_FILE> -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

    Note: With dd-java-agent.jar library versions 0.48+, if your organization is on Datadog EU site, add -Ddd.site=datadoghq.eu or set DD_SITE=datadoghq.eu as environment variable.

  3. After a minute or two, visualize your profiles on the Datadog APM > Profiling page.

Note:

  • The -javaagent needs to be run before the -jar file, adding it as a JVM option, not as an application argument. For more information, see the Oracle documentation:

    # Good:
    java -javaagent:dd-java-agent.jar ... -jar my-service.jar -more-flags
    # Bad:
    java -jar my-service.jar -javaagent:dd-java-agent.jar ...
  • Because profiles are sent directly to Datadog without using the Datadog Agent, you must pass a valid Datadog API key.

  • As an alternative to passing arguments, you can use environment variable to set those parameters:

ArgumentsEnvironment variableDescription
-Ddd.profiling.enabledDD_PROFILING_ENABLEDSet to true to enable profiling.
-Ddd.profiling.api-key-fileDD_PROFILING_API_KEY_FILEFile that should contain the API key as a string.
DD_PROFILING_API_KEYDatadog API key.
-Ddd.siteDD_SITEDestination site for your profiles (versions 0.48+). Valid options are datadoghq.com for Datadog US site (default), and datadoghq.eu for the Datadog EU site.

The Datadog Profiler requires Python 2.7+. Memory profiling only works on Python 3.5+. To begin profiling applications:

  1. Install ddtrace with the profile flavor, which contains both tracing and profiling:

    pip install ddtrace[profiling]

    Note: Profiling is available in the ddtrace library for versions 0.36+.

  2. Add a valid Datadog API key in your environment variable: DD_PROFILING_API_KEY.

  3. Set env, service, and version as Datadog tags in your environment variables.

    export DD_PROFILING_TAGS=env:<YOUR_ENVIRONMENT>,service:<YOUR_SERVICE>,version:<YOUR_VERSION>
  4. To automatically profile your code, import ddtrace.profile.auto. After import, the profiler starts:

    import ddtrace.profiling.auto
  5. After a minute or two, visualize your profiles on the Datadog APM > Profiling page.

Note:

  • Alternatively, profile your service by running it with the wrapper pyddprofile:

    $ pyddprofile server.py
  • For advanced setup of the profiler or to add tags like service or version, use environment variables to set the parameters:

Environment variableTypeDescription
DD_API_KEYStringThe Datadog API key to use when uploading profiles. New in version 0.37.
DD_PROFILING_API_KEYStringThe Datadog API key to use when uploading profiles. Changed in 0.37: deprecated in favor of DD_API_KEY.
DD_SITEStringIf your organization is on Datadog EU site, set this to datadoghq.eu.
DD_SERVICEStringThe Datadog service name.
DD_ENVStringThe Datadog environment name, for example production, which can be set here, or in DD_PROFILING_TAGS with DD_PROFILING_TAGS="env:production".
DD_VERSIONStringThe version of your application, which can be set here, or in DD_PROFILING_TAGS with DD_PROFILING_TAGS="version:<APPLICATION_VERSION>"
DD_PROFILING_TAGSStringTags to apply to an uploaded profile. Must be a list a key:value comma separated list like: <KEY1>:<VALUE1>,<KEY2>:<VALUE2>.
Recommended for advanced usage only.
  • If you want to manually control the lifecycle of the profiler, use the ddtrace.profiling.profiler.Profiler object:

    from ddtrace.profiling import Profiler
    
    prof = Profiler()
    prof.start()
    
    # At shutdown
    prof.stop()

The Datadog Profiler requires Go 1.12+. To begin profiling applications:

  1. Get dd-trace-go using the command:

    go get gopkg.in/DataDog/dd-trace-go.v1/profiler

    Note: Profiling is available in the dd-trace-go library for versions 1.23.0+.

  2. Import the profiler at the start of your application:

    import "gopkg.in/DataDog/dd-trace-go.v1/profiler"
  3. To profile your code, add a Datadog API key, set your environment, service, and version, then start the profiler:

    err := profiler.Start(
        profiler.WithAPIKey("<DATADOG_API_KEY>")
        profiler.WithService("<SERVICE_NAME>"),
        profiler.WithEnv("<ENVIRONMENT>"),
        profiler.WithTags("version:<APPLICATION_VERSION>"),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer profiler.Stop()
  4. After a minute or two, visualize your profiles in the Datadog APM > Profiling page.

Profiler configuration:

profiler methodTypeDescription
WithAPIKeyStringThe Datadog Datadog API key
WithServiceStringThe Datadog service name, for example my-web-app, which can be set here, or in DD_TAGS.
WithEnvStringThe Datadog environment name, for example production, which can be set here, or in DD_TAGS.
WithTagsStringThe tags to apply to an uploaded profile. Must be a list of in the format <KEY1>:<VALUE1>,<KEY2>:<VALUE2>.

Profiles

Once profiling is set up, profiles are available in the APM > Profiling page within Datadog:

Each row is a profile of a process for a short amount of time. By default profiles are uploaded once a minute. Depending on the language, these processes are profiled between 15s and 60s.

You can filter according to infrastructure tags or application tags set up from your environment tracing configuration. By default the following facets are available:

FacetDefinition
EnvThe environment your application is running on (prod, staging).
ServiceThe name of the service your code is running.
VersionThe version of your code.
HostThe host name your profiled process is running on. If running in Kubernetes, it’s the name of the pod.
RuntimeThe type of runtime the profiled process is running (JVM, CPython).
LanguageThe language of your code (Java, Python).

The following measures are available:

MeasureDefinition
CPU coresCPU load summed on all cores of the process. This metric can go above 100%, and the theoretical limit is nCores * 100%.
Memory AllocationMemory allocation rate in Bytes/s over the course of the profile. This value can be above the maximum amount of RAM memory because it can be garbage collected during the profile.

Profile

Click on a line to view a specific profile:

The profile header contains information associated with your profile, like the service that generated it, or the environment and code version associated to it.

Four tabs are below the profile header:

TabDefinition
ProfilesFlame graph and summary table of the profile you are looking at. You can switch between multiple profile types (CPU, Memory allocation)
AnalysisA set of heuristics that suggest potential issues or areas of improvement in your code
MetricsProfiling metrics coming from all profiles of the same service
Runtime InfoText dump of all the runtime properties

Note: In the upper right corner of each profile, there are options to:

  • Download the profile
  • Switch the profile to full screen

Profile types

In the Profiles tab you can see all profile types available for a given language. Depending on the language, the information collected about your profile differs.

Once enabled, the following profile types are collected:

Profile typeDefinition
CPU in Java CodeShows the time each method spent running on the CPU. It includes JVM bytecode, but not native code called from within the JVM.
AllocationShows the amount of heap memory allocated by each method, including allocations which were subsequently freed.
Wall Time in Native CodeShows the elapsed time spent in native code. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the method is running. This profile does not include time spent running JVM bytecode, which is typically most of your application code.
Class loadShows the number of classes loaded by each method.
ErrorShows the number of errors thrown by each method.
File I/OShows the time each method spent reading and writing files.
LockShows the time each method spent waiting for a lock.
Socket I/OShows the time each method spent handling socket I/O.

Once enabled, the following profile types are collected:

Profile typeDefinition
CPUShows the time each function spent running on the CPU. It includes CPython bytecode, including native code called from within Python.
AllocationShows the amount of heap memory allocated by each function, including allocations which were subsequently freed. Only supported on Python 3.
WallShows the elapsed time used by each function. Elapsed time includes time when code is running on CPU, waiting for I/O, and anything else that happens while the function is running.
ExceptionsShows the number of caught or uncaught exceptions raised by each function.
LockShows the time each function spent waiting for a lock.

Once enabled, the following profile types are collected:

Profile typeDefinition
CPUShows the time each function spent running on the CPU.
AllocationShows the amount of heap memory allocated by each function, including allocations which were subsequently freed. Go calls this alloc_space. This is useful for investigating garbage collection load.
Allocation CountShows the number of objects allocated in heap memory by each function, including allocations which were subsequently freed. This is useful for investigating garbage collection load.
HeapShows the amount of heap memory allocated by each function that remained allocated. Go calls this inuse_space. This is useful for investigating the overall memory usage of your service.
Heap CountShows the number of objects allocated in heap memory by each function and that remained allocated. This is useful for investigating the overall memory usage of your service.

Troubleshooting

In case you have done all the necessary configuration steps and do not see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

  • OS type and version (e.g Linux Ubuntu 14.04.3)
  • Runtime type, version and vendor (e.g Java OpenJDK 11 AdoptOpenJDK)

Further Reading