--- title: Troubleshooting the Java Profiler description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Continuous Profiler > Profiler Troubleshooting > Troubleshooting the Java Profiler --- # Troubleshooting the Java Profiler ## Profiler engines and profile types{% #profiler-engines-and-profile-types %} The profiler automatically configures the optimal engine and settings for your environment. Use the following sections to understand the available profile types, customize their configuration, and troubleshoot common issues. ### Profiler engines{% #profiler-engines %} The Java profiler uses two different profiling engines depending on your operating system: | Engine | Platforms | | ------------------------------ | -------------------------------- | | **Datadog Profiler** | Linux | | **Java Flight Recorder (JFR)** | Windows, macOS, Linux (fallback) | - **Datadog Profiler**: The default profiling engine on Linux. Starting with dd-trace-java 1.55.0, the Datadog profiler uses its own stable stack-walker implementation for improved robustness across JVMs and GC modes. On Linux, it uses the kernel perf subsystem for accurate CPU sampling and provides the full set of profile types including wallclock profiling for latency analysis. - **Java Flight Recorder (JFR)**: Uses the JVM's built-in flight recorder. Available on all platforms but with fewer profile types. On Linux, used as a fallback when the Datadog Profiler cannot be enabled. ### CPU profiling{% #cpu-profiling %} Datadog CPU profiling is scheduled through perf events and is more accurate than JFR CPU profiling. | Engine | Environment Variable | System Property | JMC Event | Notes | | -------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | ------------------------- | --------------------------------------- | | **Datadog** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true``DD_PROFILING_DDPROF_CPU_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true``-Ddd.profiling.ddprof.cpu.enabled=true` | `datadog.ExecutionSample` | Default on Linux | | **JFR** | `DD_PROFILING_DDPROF_CPU_ENABLED=false` | `-Ddd.profiling.ddprof.cpu.enabled=false` | `jdk.ExecutionSample` | To switch from Datadog to JFR (v1.7.0+) | ### Wallclock{% #wallclock %} The Datadog profiler wallclock engine: - Samples all threads, on- or off-CPU, with active tracing activity - Can be used to diagnose trace or span latency - Is useful for profiling latency and integrates tightly with APM tracing - Enabled by default - Is specific to the Datadog profiler and is not available on Windows | Configuration | Environment Variable | System Property | JMC Event | | ------------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ---------------------- | | **Enable** (default in v1.7.0+) | `DD_PROFILING_DDPROF_ENABLED=true``DD_PROFILING_DDPROF_WALL_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true``-Ddd.profiling.ddprof.wall.enabled=true` | `datadog.MethodSample` | ### Allocation profiling{% #allocation-profiling %} The Datadog allocation profiling engine: - Contextualizes allocation profiles with endpoint information - Supports allocation profiles filtered by endpoint - Requires Java 11+; automatically disabled on JDK versions with unstable JVMTI APIs (below 11.0.23, 17.0.11, 21.0.3) to prevent stability issues - For JMC users, the Datadog allocation events are `datadog.ObjectAllocationInNewTLAB` and `datadog.ObjectAllocationOutsideTLAB` The JFR-based allocation profiling engine: - Enabled by default on JDK 16+ - Not enabled by default on lower JDKs (8-15), as allocation-intensive applications can lead to high overhead and large recording sizes. - For JMC users, the JFR allocation events are `jdk.ObjectAllocationInNewTLAB` and `jdk.ObjectAllocationOutsideTLAB` for Java 8+, `jdk.ObjectAllocationSample` for Java 16+ | Engine | Environment Variable | System Property | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | **Datadog** (default on Java 11.0.23+, 17.0.11+, 21.0.3+, 22+) | `DD_PROFILING_DDPROF_ENABLED=true``DD_PROFILING_DDPROF_ALLOC_ENABLED=true` | `-Ddd.profiling.ddprof.enabled=true``-Ddd.profiling.ddprof.alloc.enabled=true` | | **JFR** (Java 8+) | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB` | | **JFR** (Java 16+) | `DD_PROFILING_ENABLED_EVENTS=jdk.ObjectAllocationSample` | `-Ddd.profiling.enabled.events=jdk.ObjectAllocationSample` | Alternatively, you can enable the following events in your `jfp` override template file: ``` #Java 8+ jdk.ObjectAllocationInNewTLAB#enabled=true jdk.ObjectAllocationOutsideTLAB#enabled=true #Java 16+ jdk.ObjectAllocationSample#enabled=true ``` ### Live heap profiling{% #live-heap-profiling %} Live heap profiling tracks which allocated objects survive garbage collection, making it useful for diagnosing memory leaks and understanding overall heap usage. Starting with dd-trace-java **v1.61.0**, it is **enabled by default** on Linux. The profiler automatically selects the most accurate engine available for your JVM version. The live-heap profiler: - Useful for investigating the overall memory usage of your service and identifying potential memory leaks - Samples allocations and tracks whether those samples survived the most recent garbage collection cycle - Automatically uses the Datadog profiler engine on JDK 11.0.23+, 17.0.11+, 21.0.3+, or 22+, and falls back to the JFR engine (`jdk.OldObjectSample`) on older supported JDK versions - Not available on Windows - For JMC users, the Datadog live-heap event is `datadog.HeapLiveObject`; the JFR fallback emits `jdk.OldObjectSample` To disable live heap profiling entirely (both Datadog engine and JFR fallback): | Environment Variable | System Property | | --------------------------------- | ----------------------------------- | | `DD_PROFILING_HEAP_ENABLED=false` | `-Ddd.profiling.heap.enabled=false` | To disable only the Datadog native engine implementation (the JFR fallback `jdk.OldObjectSample` remains active on older JDKs): | Environment Variable | System Property | | -------------------------------------------- | ---------------------------------------------- | | `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED=false` | `-Ddd.profiling.ddprof.liveheap.enabled=false` | ### Heap histogram metrics{% #heap-histogram-metrics %} {% alert level="info" %} This feature requires at least Java 17.0.9 or newer and does not work with ZGC {% /alert %} To enable the heap histogram metrics, start your application with one of the following: | Environment Variable | System Property | | ------------------------------------------ | -------------------------------------------- | | `DD_PROFILING_HEAP_HISTOGRAM_ENABLED=true` | `-Ddd.profiling.heap.histogram.enabled=true` | This data is collected when the JVM performs a Full Garbage Collection cycle and may only appear intermittently or not at all if your service does not have significant memory pressure. For more information, see [Diagnosing and solving memory leaks](https://docs.datadoghq.com/profiler/guide/solve-memory-leaks.md). ### Trace to profiling integration{% #trace-to-profiling-integration %} The [Trace to Profiling integration](https://docs.datadoghq.com/profiler/connect_traces_and_profiles.md#identify-code-hotspots-in-slow-traces) identifies code hotspots in slow traces. The following minimum versions are required: | JDK | Minimum version | dd-trace-java version | | ------- | ---------------------- | --------------------- | | OpenJDK | 8u352, 11.0.17, 17.0.5 | 1.17.0+ | | OpenJ9 | 8u362, 11.0.18, 17.0.6 | 1.17.0+ | ### Configuration reference{% #configuration-reference %} The following settings allow fine-grained control over the profiler engines. These are typically not needed for standard use cases. For detailed information about each profiler type, see the corresponding sections above: CPU profiling, Wallclock, Allocation profiling, and Live heap profiling. **Note**: - `DD_PROFILING_HEAP_ENABLED` controls all live heap profiling. Setting it to `false` disables both the Datadog native engine and the JFR `jdk.OldObjectSample` fallback. - `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED` only controls the Datadog native engine. The JFR fallback is unaffected by this setting. | Environment variable | System property | Description | | -------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `DD_PROFILING_HEAP_ENABLED` | `-Ddd.profiling.heap.enabled` | Main toggle for all live heap profiling, including both the Datadog native engine and the JFR `jdk.OldObjectSample` fallback (default: true on supported platforms since v1.61.0) | | `DD_PROFILING_DDPROF_ENABLED` | `-Ddd.profiling.ddprof.enabled` | Enable the Datadog profiler engine (Linux only, default: true since v1.7.0) | | `DD_PROFILING_DDPROF_CPU_ENABLED` | `-Ddd.profiling.ddprof.cpu.enabled` | Enable CPU profiling with the Datadog engine | | `DD_PROFILING_DDPROF_WALL_ENABLED` | `-Ddd.profiling.ddprof.wall.enabled` | Enable wallclock profiling (default: true since v1.7.0) | | `DD_PROFILING_DDPROF_ALLOC_ENABLED` | `-Ddd.profiling.ddprof.alloc.enabled` | Enable allocation profiling with the Datadog engine | | `DD_PROFILING_DDPROF_LIVEHEAP_ENABLED` | `-Ddd.profiling.ddprof.liveheap.enabled` | Override for the Datadog native engine live heap implementation only; does not affect the JFR fallback (default: enabled on supported JDK versions (11.0.23+, 17.0.11+, 21.0.3+, or 22+), requires `DD_PROFILING_HEAP_ENABLED=true`) | | `DD_PROFILING_ENABLED_EVENTS` | `-Ddd.profiling.enabled.events` | Enable specific JFR events (for example: `jdk.ObjectAllocationInNewTLAB,jdk.ObjectAllocationOutsideTLAB`) | ### JDK Mission Control (JMC) event reference{% #jdk-mission-control-jmc-event-reference %} If you are analyzing profiles with JDK Mission Control, the following table provides a reference for events emitted by the profiler. | Profile type | JFR event | Datadog event | | ------------ | ------------------------------------------------------------------ | -------------------------------------------------------------------------- | | CPU | `jdk.ExecutionSample` | `datadog.ExecutionSample` | | Wallclock | - | `datadog.MethodSample` | | Allocation | `jdk.ObjectAllocationInNewTLAB`, `jdk.ObjectAllocationOutsideTLAB` | `datadog.ObjectAllocationInNewTLAB`, `datadog.ObjectAllocationOutsideTLAB` | | Live heap | `jdk.OldObjectSample` | `datadog.HeapLiveObject` | ## Advanced configuration{% #advanced-configuration %} ### Reduce overhead from default setup{% #reduce-overhead-from-default-setup %} If the default setup overhead is not acceptable, you can use the profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default: - Increases sampling threshold to 500ms for `ThreadSleep`, `ThreadPark`, and `JavaMonitorWait` events compared to 100ms default - Disables `ObjectAllocationInNewTLAB`, `ObjectAllocationOutsideTLAB`, `ExceptionSample`, `ExceptionCount` events To use the minimal configuration, verify you have a recent version of the Java tracer, then change the service invocation to the following: ``` java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=minimal -jar .jar ``` ### Increase profiler information granularity{% #increase-profiler-information-granularity %} If you want more granularity in your profiling data, you can specify the `comprehensive` configuration. This approach increases your profiler overhead at the cost of further granularity. Comprehensive configuration has the following changes compared to the default: - Reduces sampling threshold to 10ms for `ThreadSleep`, `ThreadPark`, and `JavaMonitorWait` events compared to 100ms default - Enables `ObjectAllocationInNewTLAB`, `ObjectAllocationOutsideTLAB`, `ExceptionSample`, `ExceptionCount` events To use the comprehensive configuration, verify you have a recent version of the Java tracer, then change your service invocation to the following: ``` java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=comprehensive -jar .jar ``` ### Removing sensitive information from profiles{% #removing-sensitive-information-from-profiles %} If your system properties contain sensitive information such as user names or passwords, turn off the system property event by creating a `jfp` override template file with `jdk.InitialSystemProperty` disabled: ``` jdk.InitialSystemProperty#enabled=false ``` See following section to learn how to use override templates ### Creating and using a JFR template override file{% #creating-and-using-a-jfr-template-override-file %} Override templates let you specify profiling properties to override. However, the default settings are balanced for a good tradeoff between overhead and data density that cover most use cases. To use an override file, perform the following steps: 1. Create an override file in a directory accessible by `dd-java-agent` at service invocation: ``` touch dd-profiler-overrides.jfp ``` 1. Add your desired overrides to the jfp file. For example, to disable allocation profiling and JVM system properties, your `dd-profiler-overrides.jfp` file would look like the following: ``` jdk.ObjectAllocationInNewTLAB#enabled=false jdk.ObjectAllocationOutsideTLAB#enabled=false jdk.InitialSystemProperty#enabled=false ``` 1. When running your application with `dd-java-agent`, your service invocation must point to the override file with `-Ddd.profiling.jfr-template-override-file=`, for example: ``` java -javaagent:/path/to/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.profiling.jfr-template-override-file= -jar path/to/your/app.jar ``` ### Advanced Linux settings for CPU profiles{% #advanced-linux-settings-for-cpu-profiles %} The CPU profiler engine works on most systems, but if the value of `/proc/sys/kernel/perf_event_paranoid` is set to `3`, the profiler can't use perf events to schedule CPU sampling. This results in degraded profile quality, falling back to using itimer. Set `/proc/sys/kernel/perf_event_paranoid` to `2` or lower with the following command: ```shell sudo sh -c 'echo 2 >/proc/sys/kernel/perf_event_paranoid' ``` ## Troubleshooting{% #troubleshooting %} Find solutions for common issues with the Java profiler, including missing profiles, application crashes, and performance overhead. ### Missing profiles in the profile search page{% #missing-profiles-in-the-profile-search-page %} If you've configured the profiler and don't see profiles in the profile search page, turn on [debug mode](https://docs.datadoghq.com/tracing/troubleshooting.md#debugging-and-logging) and [open a support ticket](https://docs.datadoghq.com/help/) with debug files and the following information: - Operating system type and version (for example, Linux Ubuntu 20.04) - Runtime type, version, and vendor (for example, Java OpenJDK 11 AdoptOpenJDK) ### Application crashes{% #application-crashes %} If the profiler causes your application to crash, [open a support ticket](https://docs.datadoghq.com/help/) with the crash logs and the following information: - Operating system type and version (for example, Linux Ubuntu 20.04) - Runtime type, version, and vendor (for example, Java OpenJDK 11 AdoptOpenJDK) - Application name and tracer version (for example, dd-trace-java 1.55.0) ### High overhead{% #high-overhead %} If you notice increased CPU usage, memory consumption, or application latency after enabling the profiler, [open a support ticket](https://docs.datadoghq.com/help/) with the following information: - Operating system type and version (for example, Linux Ubuntu 20.04) - Runtime type, version, and vendor (for example, Java OpenJDK 11 AdoptOpenJDK) - Application name and tracer version (for example, dd-trace-java 1.55.0) - Description of the overhead observed (for example, CPU increase, memory growth, latency impact) - Profiler configuration settings, including any non-default options ### Large allocation events overwhelming the profiler{% #large-allocation-events-overwhelming-the-profiler %} To turn off allocation profiling, disable the following events in your `jfp` override template file: ``` jdk.ObjectAllocationInNewTLAB#enabled=false jdk.ObjectAllocationOutsideTLAB#enabled=false ``` Learn how to use override templates. ### Memory leak detection slowing down garbage collector{% #memory-leak-detection-slowing-down-garbage-collector %} {% tab title="JFR" %} To turn off memory leak detection, disable the following event in your `jfp` override template file: ``` jdk.OldObjectSample#enabled=false ``` Learn how to use override templates. {% /tab %} {% tab title="Datadog Profiler" %} If you are using the alpha feature of live heap profiling, you can tune the overhead by changing the percentage of the tracked allocation samples. ```shell # track only 10% of the allocation samples java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.ddprof.liveheap.enabled=true -Ddd.profiling.ddprof.liveheap.sample_percent=10 -jar .jar ``` {% /tab %} ### Exceptions overwhelming the profiler{% #exceptions-overwhelming-the-profiler %} The Datadog exception profiler has a small footprint and overhead under normal conditions. If a lot of exceptions are created and thrown, it can cause significant overhead for the profiler. This can happen when you use exceptions for control flow. If you have an unusually high exception rate, turn off exception profiling temporarily until you fix the cause. To disable exception profiling, start the tracer with the `-Ddd.integration.throwables.enabled=false` JVM setting. Note: Turn this setting back on after you've returned to a more typical rate of exceptions. ### PODs are getting evicted due to disk usage{% #pods-are-getting-evicted-due-to-disk-usage %} The profiler uses ephemeral storage (usually `/tmp`) to save captured profiling data. If the node is under disk pressure and the pod hasn't requested ephemeral storage, it may be evicted. Fix: Add a small ephemeral storage request (such as 100MB) in the pod spec to prevent eviction. ### Managing issues related to the tmp folder{% #managing-issues-related-to-the-tmp-folder %} The Continuous Profiler may encounter errors related to the use of the system `/tmp` directory, particularly in environments with strict security or limited execution permissions (for example, Docker, Kubernetes, or SELinux-enabled systems). These issues can lead to: - Profiling startup failures - Inability to load native `.so` libraries - Accumulation of stale temp files across JVM restarts or crashes Below are basic troubleshooting steps for resolving those issues: - Use dd-trace-java Version 1.47.0 or later Starting with v1.47.0, the profiler uses PID-specific subdirectories inside the configured temp directory. This reduces clutter and potential conflicts from orphaned files when JVM processes exit unexpectedly. - Specify a custom executable temp directory To ensure proper operation across environments, explicitly configure a writable, executable temp directory using the following JVM option: ``` -Ddd.profiling.tempdir= ``` Directory Requirements: - Must be writable by the JVM process - Must have execute permissions on all levels of the path - Must comply with SELinux policies, if enforced Example: ``` mkdir -p /opt/datadog-profiler-tmp chmod 755 /opt/datadog-profiler-tmp java -Ddd.profiling.tempdir=/opt/datadog-profiler-tmp -javaagent:/path/to/dd-java-agent.jar ... ``` - If you enable profiling using SSI, you can include the below environment variable in the `application_monitoring.yaml`. ``` DD_PROFILING_TEMPDIR: ``` ### Java 8 support{% #java-8-support %} The following OpenJDK 8 vendors are supported for Continuous Profiling because they include JDK Flight Recorder in their latest versions: | Vendor | JDK version that includes Flight Recorder | | --------------------------- | ----------------------------------------- | | Azul | u212 (u262 is recommended) | | AdoptOpenJDK | u262 | | RedHat | u262 | | Amazon (Corretto) | u262 | | Bell-Soft (Liberica) | u262 | | All vendors upstream builds | u272 | If your vendor is not on the list, [open a support ticket](https://docs.datadoghq.com/help/), as other vendors may be in development or available in Preview support. ## Further Reading{% #further-reading %} - [APM Troubleshooting](https://docs.datadoghq.com/tracing/troubleshooting.md)