Missing profiles in the profile search page

If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

• Operating system type and version (for example, Linux Ubuntu 20.04)
• Runtime type, version, and vendor (for example, Java OpenJDK 11 AdoptOpenJDK)

Reduce overhead from default setup

If the default setup overhead is not acceptable, you can use the profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default:

• Increases sampling threshold to 500ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
• Disables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the minimal configuration ensure you have dd-java-agent version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=minimal -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Modify the maximum stack depth for collected stack traces

If the default maximum stack depth of 512 is not sufficient for your use case, or it is causing performance issues, you can increase it by setting the dd.profiling.stackdepth system property.

For example, to decrease the maximum stack depth to 256, start your service with the following JVM setting:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.stackdepth=256 -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

The same limit will be used for data collected by JFR and the Datadog profiler.

Note: If the -XX:FlightRecorderOptions=stackdepth=<stack-depth> JVM argument is provided, the maximum stack depth set via the dd.profiling.stackdepth system property will be ignored in the data collected by JFR, for technical reasons.

Increase profiler information granularity

If you want more granularity in your profiling data, you can specify the comprehensive configuration. Note that this approach will increase your profiler overhead at the cost of further granularity. Comprehensive configuration has the following changes compared to the default:

• Reduces sampling threshold to 10ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
• Enables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the comprehensive configuration ensure you have dd-trace-java version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=comprehensive -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Enabling the allocation profiler

On Java 15 and lower, the allocation profiler is turned off by default because it can overwhelm the profiler in allocation-heavy applications.

To enable the allocation profiler, start your application with the -Ddd.profiling.allocation.enabled=true JVM setting or the DD_PROFILING_ALLOCATION_ENABLED=true environment variable.

Alternatively, you can enable the following events in your jfp override template file:

jdk.ObjectAllocationInNewTLAB#enabled=true
jdk.ObjectAllocationOutsideTLAB#enabled=true

Learn how to use override templates.

Enabling the heap profiler

Alternatively, you can enable the following events in your jfp override template file:

jdk.OldObjectSample#enabled=true

Learn how to use override templates.

Removing sensitive information from profiles

If your system properties contain sensitive information such as user names or passwords, turn off the system property event by creating a jfp override template file with jdk.InitialSystemProperty disabled:

jdk.InitialSystemProperty#enabled=false

Learn how to use override templates.

Large allocation events overwhelming the profiler

To turn off allocation profiling, disable the following events in your jfp override template file:

jdk.ObjectAllocationInNewTLAB#enabled=false
jdk.ObjectAllocationOutsideTLAB#enabled=false

Learn how to use override templates.

Memory leak detection slowing down garbage collector

To turn off memory leak detection, disable the following event in your jfp override template file:

jdk.OldObjectSample#enabled=false

Learn how to use override templates.

Exceptions overwhelming the profiler

The Datadog exception profiler has a small footprint and overhead under normal conditions. If a lot of exceptions are created and thrown, it can cause significant overhead for the profiler. This can happen when you use exceptions for control flow. If you have an unusually high exception rate, turn off exception profiling temporarily until you fix the cause.

To disable exception profiling, start the tracer with the -Ddd.integration.throwables.enabled=false JVM setting.

Remember to turn this setting back on after you’ve returned to a more typical rate of exceptions.

Java 8 support

The following OpenJDK 8 vendors are supported for Continuous Profiling because they include JDK Flight Recorder in their latest versions:

If your vendor is not on the list, open a support ticket, as other vendors may be in development or available for beta support.

Creating and using a JFR template override file

Override templates let you specify profiling properties to override. However, the default settings are balanced for a good tradeoff between overhead and data density that cover most use cases. To use an override file, perform the following steps:

1. Create an override file in a directory accessible by dd-java-agent at service invocation:

   touch dd-profiler-overrides.jfp
   

2. Add your desired overrides to the jfp file. For example, to disable allocation profiling and JVM system properties, your dd-profiler-overrides.jfp file would look like the following:

   jdk.ObjectAllocationInNewTLAB#enabled=false
   jdk.ObjectAllocationOutsideTLAB#enabled=false
   jdk.InitialSystemProperty#enabled=false
   

3. When running your application with dd-java-agent, your service invocation must point to the override file with -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp>, for example:

   java -javaagent:/path/to/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp> -jar path/to/your/app.jar
   

Further Reading

お役に立つドキュメント、リンクや記事:

APM TroubleshootingDOCUMENTATION