Enabling the Java Profiler

The profiler is shipped within Datadog tracing libraries. If you are already using APM to collect traces for your application, you can skip installing the library and go directly to enabling the profiler.

Requirements

For a summary of the minimum and recommended runtime and tracer versions across all languages, read Supported Language and Tracer Versions.

As of dd-trace-java 1.0.0, you have two options for the engine that generates profile data for Java applications: Java Flight Recorder (JFR) or the Datadog Profiler. As of dd-trace-java 1.7.0, Datadog Profiler is the default. Each profiler engine has different side effects, requirements, available configurations, and limitations, and this page describes each. You can enable either one or both engines. Enabling both captures both profile types at the same time.

    Supported operating systems:

    • Linux

    Minimum JDK versions:

    • OpenJDK 8u352+, 11.0.17+, 17.0.5+ (including builds on top of it: Amazon Corretto, Azul Zulu, and others)
    • Oracle JDK 8u352+, 11.0.17+, 17.0.5+
    • OpenJ9 JDK 8u372+, 11.0.18+, 17.0.6+ (used on Eclipse OpenJ9, IBM JDK, IBM Semeru Runtime). The profiler is disabled by default for OpenJ9 due to the possibility of crashing JVM caused by a subtle bug in JVTMI implementation. If you are not experiencing any crashes, you can enable the profiler by adding -Ddd.profiling.ddprof.enabled=true.
    • Azul Platform Prime 23.05.0.0+ (formerly Azul Zing)

    The Datadog Profiler uses the JVMTI AsyncGetCallTrace function, in which there is a known issue prior to JDK release 17.0.5. This fix was backported to 11.0.17 and 8u352. The Datadog Profiler is not enabled unless the JVM the profiler is deployed into has this fix. Upgrade to at least 8u352, 11.0.17, 17.0.5, or the latest non-LTS JVM version to use the Datadog Profiler.

    Supported operating systems:

    • Linux
    • Windows

    Minimum JDK versions:

    • OpenJDK 1.8.0.262/8u262+, 11+ (including builds on top of it: Amazon Corretto, and others)
    • Oracle JDK 11+ (Enabling the JFR may require a commercial license from Oracle. Reach out to your Oracle representative to confirm whether this is part of your license)
    • Azul Zulu 8 (version 1.8.0.212/8u212+), 11+
    • GraalVM 17+ - both, JIT and AOT (native-image) versions

    Because non-LTS JDK versions may not contain stability and performance fixes related to the Datadog Profiler library, use versions 8, 11, and 17 of the Long Term Support JDK.

    Additional requirements for profiling Code Hotspots:

    • OpenJDK 11+ and dd-trace-java version 0.65.0+
    • OpenJDK 8 8u282+ and dd-trace-java version 0.77.0+

    All JVM-based languages, such as Java, Scala, Groovy, Kotlin, and Clojure are supported.

    Continuous Profiler is not supported on serverless platforms, such as AWS Lambda.

    Installation

    To begin profiling applications:

    1. Ensure Datadog Agent v6+ is installed and running. Datadog recommends using Datadog Agent v7+. If you don’t have APM enabled to set up your application to send data to Datadog, in your Agent, set the DD_APM_ENABLED environment variable to true and listening to the port 8126/TCP.

    2. Download dd-java-agent.jar, which contains the Java Agent class files:

        wget -O dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'

        curl -Lo dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'

        ADD 'https://dtdg.co/latest-java-tracer' dd-java-agent.jar

        Note: Profiler is available in the dd-java-agent.jar library in versions 0.55+.

      • Enable the profiler by setting -Ddd.profiling.enabled flag or DD_PROFILING_ENABLED environment variable to true. Specify dd.service, dd.env, and dd.version so you can filter and group your profiles across these dimensions:

          Invoke your service:

          java \
    -javaagent:dd-java-agent.jar \
    -Ddd.service=<YOUR_SERVICE> \
    -Ddd.env=<YOUR_ENVIRONMENT> \
    -Ddd.version=<YOUR_VERSION> \
    -Ddd.profiling.enabled=true \
    -XX:FlightRecorderOptions=stackdepth=256 \
    -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

          export DD_SERVICE=<YOUR_SERVICE>
export DD_ENV=<YOUR_ENV>
export DD_VERSION=<YOUR_VERSION>
export DD_PROFILING_ENABLED=true
java \
    -javaagent:dd-java-agent.jar \
    -XX:FlightRecorderOptions=stackdepth=256 \
    -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

        Follow the Tracer Setup Instructions to build your native image with the Datadog Java Profiler.

        When the service binary is built, you can use environment variables to enable and configure the Datadog Java Profiler:

        DD_PROFILING_ENABLED=true DD_PROFILING_DIRECTALLOCATION_ENABLED=true ./my_service

        Note: Only JFR-based profiling is supported for the GraalVM native-image applications. None of the DDPROF related configuration options are effective.

        Note: The -javaagent argument needs to be before -jar. This adds it as a JVM option rather than an application argument. For example, java -javaagent:dd-java-agent.jar ... -jar my-service.jar -more-flags. For more information, see the Oracle documentation.

        1. After a minute or two, you can visualize your profiles on the Datadog APM > Profiling page.

        Enabling CPU profiler engine options

        Since dd-trace-java version 1.5.0, you have two options for the CPU profiler used, Datadog or Java Flight Recorder (JFR). Since version 1.7.0, Datadog is the default, but you can also optionally enable JFR for CPU profiling. You can enable either one or both engines. Enabling both captures both profile types at the same time.

        The Datadog profiler records the active span on every sample, which improves the fidelity of the Code Hotspots and Endpoint profiling features. Enabling this engine supports much better integration with APM tracing.

        The Datadog profiler consists of several profiling engines, including CPU, wallclock, allocation, and memory leak profilers.

          Requires JDK 11+.

          The Datadog profiler is enabled by default in dd-trace-java versions 1.7.0+. Datadog CPU profiling is scheduled through perf events and is more accurate than JFR CPU profiling. To enable CPU profiling:

          export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+
export DD_PROFILING_DDPROF_CPU_ENABLED=true

          or:

          -Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+
-Ddd.profiling.ddprof.cpu.enabled=true

          For JDK Mission Control (JMC) users, the Datadog CPU sample event is datadog.ExecutionSample.

          Linux settings

          The CPU engine works on most systems, but if the value of /proc/sys/kernel/perf_event_paranoid is set to 3, the profiler can’t use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using itimer. Set /proc/sys/kernel/perf_event_paranoid to 2 or lower with the following command:

          sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid'

          For version 1.7.0+, to switch from the default Datadog to JFR CPU profiling:

          export DD_PROFILING_DDPROF_CPU_ENABLED=false

          or:

          -Ddd.profiling.ddprof.cpu.enabled=false

          For JDK Mission Control (JMC) users, the JFR CPU sample event is jdk.ExecutionSample.

          Datadog profiler wallclock engine

          The wallclock profiling engine is useful for profiling latency and integrates tightly with APM tracing. The engine samples all threads, on- or off-CPU, with active tracing activity and can be used to diagnose trace or span latency. The engine has been enabled by default since 1.7.0.

          -Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+
-Ddd.profiling.ddprof.wall.enabled=true

          For version 1.7.0+, to disable the wallclock profiler:

          export DD_PROFILING_DDPROF_WALL_ENABLED=false

          or:

          -Ddd.profiling.ddprof.wall.enabled=false

          For JMC users, the datadog.MethodSample event is emitted for wallclock samples.

          The wallclock engine does not depend on the /proc/sys/kernel/perf_event_paranoid setting.

          Profiler allocation engine

            The JFR based allocation profiling engine is enabled by default since JDK 16. The reason it’s not enabled by default for JDK 8 and 11, is that an allocation intensive application can lead to high overhead and large recording sizes. To enable it for JDK 8 and 11, add the following:

            export DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB

            or:

            -Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB

            The Datadog allocation profiling engine contextualizes allocation profiles, which supports allocation profiles filtered by endpoint. In dd-java-agent earlier than v1.17.0 it is disabled by default. Enable it with:

            export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+
export DD_PROFILING_DDPROF_ALLOC_ENABLED=true # this is the default in v1.17.0+

            or:

            -Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+
-Ddd.profiling.ddprof.alloc.enabled=true # this is the default in v1.17.0+

            For JMC users, the Datadog allocation events are datadog.ObjectAllocationInNewTLAB and datadog.ObjectAllocationOutsideTLAB.

            The allocation profiler engine does not depend on the /proc/sys/kernel/perf_event_paranoid setting.

            Live-heap profiler engine (alpha)

            Since: v1.17.0. Requires JDK 11+.

            This is an alpha feature, it is not recommended to enable this feature in production environments.

            The live-heap profiler engine is useful for investigating the overall memory usage of your service and identifying potential memory leaks. The engine samples allocations and keeps track of whether those samples survived the most recent garbage collection cycle. The number of surviving samples is used to estimate the number of live objects in the heap. The number of tracked samples is limited to avoid unbounded growth of the profiler’s memory usage.

            The engine is disabled by default, but you can enable it with:

            export DD_PROFILING_DDPROF_LIVEHEAP_ENABLED=true

            or:

            -Ddd.profiling.ddprof.liveheap.enabled=true

            For JMC users, the Datadog live-heap event is datadog.HeapLiveObject.

            The allocation engine does not depend on the /proc/sys/kernel/perf_event_paranoid setting.

            Collecting native stack traces

            If the Datadog profiler CPU or wallclock engines are enabled, you can collect native stack traces. Native stack traces include things like JVM internals, native libraries used by your application or the JVM, and syscalls.

            Native stack traces are not collected by default because usually they do not provide actionable insights and walking native stacks can potentially impact application stability. Test this setting in a non-production environment before you try using it in production.

            To enable native stack trace collection, understanding that it can destabilize your application, set:

            export DD_PROFILING_DDPROF_ENABLED=true # this is the default in v1.7.0+
export DD_PROFILING_DDPROF_CSTACK=dwarf

            or:

            -Ddd.profiling.ddprof.enabled=true # this is the default in v1.7.0+
-Ddd.profiling.ddprof.cstack=dwarf

            Configuration

            You can configure the profiler using the following environment variables:

            Environment variableTypeDescription
            DD_PROFILING_ENABLEDBooleanAlternate for -Ddd.profiling.enabled argument. Set to true to enable profiler.
            DD_PROFILING_ALLOCATION_ENABLEDBooleanAlternate for -Ddd.profiling.allocation.enabled argument. Set to true to enable the allocation profiler. It requires the profiler to be enabled already.
            DD_ENVStringThe environment name, for example: production.
            DD_SERVICEStringThe service name, for example, web-backend.
            DD_VERSIONStringThe version of your service.
            DD_TAGSStringTags to apply to an uploaded profile. Must be a list of <key>:<value> separated by commas such as: layer:api, team:intake.

            Not sure what to do next?

            The Getting Started with Profiler guide takes a sample service with a performance problem and shows you how to use Continuous Profiler to understand and fix the problem.

            Further Reading

            Additional helpful documentation, links, and articles:

            Getting Started with ProfilerDOCUMENTATION more more Learn more about available profile visualizationsDOCUMENTATION more more Fix problems you encounter while using the profilerDOCUMENTATION more more