Profiler Troubleshooting

Profiler Troubleshooting

Missing profiles in the profile search page

If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

  • Operating system type and version (for example, Linux Ubuntu 14.04.3)
  • Runtime type, version, and vendor (for example, Java OpenJDK 11 AdoptOpenJDK)

Reduce overhead from default setup

If the default setup overhead is not acceptable, you can use the profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default:

  • Increases sampling threshold to 500ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
  • Disables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the minimal configuration ensure you have dd-java-agent version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=minimal -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Increase profiler information granularity

If you want more granularity in your profiling data, you can specify comprehensive configuration. Note that this approach will increase your profiler overhead at the cost of further granularity. Comprehensive configuration has the following changes compared to the default:

  • Reduces sampling threshold to 10ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
  • Enables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the comprehensive configuration ensure you have dd-trace-java version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=comprehensive -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Removing sensitive information from profiles

If your system properties contain sensitive information such as user names or passwords, turn off the system property event by creating a jfp override template file with jdk.InitialSystemProperty disabled:

example-template.jfp

jdk.InitialSystemProperty#enabled=false

Learn how to use override templates.

Large allocation events overwhelming the profiler

To turn off allocation profiling, disable the following events in your jfp override template file:

example-template.jfp

jdk.ObjectAllocationInNewTLAB#enabled=false
jdk.ObjectAllocationOutsideTLAB#enabled=false
jdk.OldObjectSample#enabled=false

Learn how to use override templates.

Memory leak detection slowing down garbage collector

To turn off memory leak detection, disable the following event in your jfp override template file:

example-template.jfp

jdk.OldObjectSample#enabled=false

Learn how to use override templates.

Exceptions overwhelming the profiler

The Datadog exception profiler has a small footprint and overhead under normal conditions. If a lot of exceptions are created and thrown, it can cause significant overhead for the profiler. This can happen when you use exceptions for control flow. If you have an unusually high exception rate, turn off exception profiling temporarily until you fix the cause.

To disable exception profiling, start the tracer with the -Ddd.integration.throwables.enabled=false JVM setting.

Remember to turn this setting back on after you’ve returned to a more typical rate of exceptions.

Java 8 support

The following OpenJDK 8 vendors are supported for Continuous Profiling because they include JDK Flight Recorder in their latest versions:

VendorJDK version that includes Flight Recorder
Azulu212 (u262 is recommended)
AdoptOpenJDKu262
RedHatu262
Amazon (Corretto)u262
Bell-Soft (Liberica)u262
All vendors upstream buildsu272

If your vendor is not on the list, open a support ticket, we can let you know if we’re planning to support it or if we already offer beta support.

Creating and using a JFR template override file

Override templates let you specify profiling properties to override. However, the default settings are balanced for a good tradeoff between overhead and data density that cover most use cases. To use an override file, perform the following steps:

  1. Create an override file in a directory accessible by dd-java-agent at service invocation:

    touch dd-profiler-overrides.jfp
    
  2. Add your desired overrides to the jfp file. For example, to disable allocation profiling and JVM system properties, your dd-profiler-overrides.jfp file would look like the following:

    example-template.jfp

        jdk.ObjectAllocationInNewTLAB#enabled=false
        jdk.ObjectAllocationOutsideTLAB#enabled=false
        jdk.InitialSystemProperty#enabled=false
        
  3. When running your application with dd-java-agent, your service invocation must point to the override file with -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp>, for example:

    example-template.jfp

        java -javaagent:/path/to/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.trace.sample.rate=1 -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp> -jar path/to/your/app.jar
        

Further Reading

Additional helpful documentation, links, and articles: