Profiler Troubleshooting

Profiler Troubleshooting

Missing profiles in the profile search page

If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

  • Operating system type and version (for example, Linux Ubuntu 20.04)
  • Runtime type, version, and vendor (for example, Java OpenJDK 11 AdoptOpenJDK)

Reduce overhead from default setup

If the default setup overhead is not acceptable, you can use the profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default:

  • Increases sampling threshold to 500ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
  • Disables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the minimal configuration ensure you have dd-java-agent version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=minimal -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Increase profiler information granularity

If you want more granularity in your profiling data, you can specify the comprehensive configuration. Note that this approach will increase your profiler overhead at the cost of further granularity. Comprehensive configuration has the following changes compared to the default:

  • Reduces sampling threshold to 10ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
  • Enables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the comprehensive configuration ensure you have dd-trace-java version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=comprehensive -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Enabling the allocation profiler

On Java 15 and lower, the allocation profiler is turned off by default because it can overwhelm the profiler in allocation-heavy applications.

To enable the allocation profiler, start your application with the -Ddd.profiling.allocation.enabled=true JVM setting or the DD_PROFILING_ALLOCATION_ENABLED=true environment variable.

Alternatively, you can enable the following events in your jfp override template file:

jdk.ObjectAllocationInNewTLAB#enabled=true
jdk.ObjectAllocationOutsideTLAB#enabled=true

Learn how to use override templates.

Enabling the heap profiler

The Java heap profiler feature is in beta.
To enable the heap profiler, start your application with the `-Ddd.profiling.heap.enabled=true` JVM setting or the `DD_PROFILING_HEAP_ENABLED=true` environment variable.

Alternatively, you can enable the following events in your jfp override template file:

jdk.OldObjectSample#enabled=true

Learn how to use override templates.

Removing sensitive information from profiles

If your system properties contain sensitive information such as user names or passwords, turn off the system property event by creating a jfp override template file with jdk.InitialSystemProperty disabled:

jdk.InitialSystemProperty#enabled=false

Learn how to use override templates.

Large allocation events overwhelming the profiler

To turn off allocation profiling, disable the following events in your jfp override template file:

jdk.ObjectAllocationInNewTLAB#enabled=false
jdk.ObjectAllocationOutsideTLAB#enabled=false

Learn how to use override templates.

Memory leak detection slowing down garbage collector

To turn off memory leak detection, disable the following event in your jfp override template file:

jdk.OldObjectSample#enabled=false

Learn how to use override templates.

Exceptions overwhelming the profiler

The Datadog exception profiler has a small footprint and overhead under normal conditions. If a lot of exceptions are created and thrown, it can cause significant overhead for the profiler. This can happen when you use exceptions for control flow. If you have an unusually high exception rate, turn off exception profiling temporarily until you fix the cause.

To disable exception profiling, start the tracer with the -Ddd.integration.throwables.enabled=false JVM setting.

Remember to turn this setting back on after you’ve returned to a more typical rate of exceptions.

Java 8 support

The following OpenJDK 8 vendors are supported for Continuous Profiling because they include JDK Flight Recorder in their latest versions:

Vendor JDK version that includes Flight Recorder
Azul u212 (u262 is recommended)
AdoptOpenJDK u262
RedHat u262
Amazon (Corretto) u262
Bell-Soft (Liberica) u262
All vendors upstream builds u272

If your vendor is not on the list, open a support ticket, we can let you know if we’re planning to support it or if we already offer beta support.

Creating and using a JFR template override file

Override templates let you specify profiling properties to override. However, the default settings are balanced for a good tradeoff between overhead and data density that cover most use cases. To use an override file, perform the following steps:

  1. Create an override file in a directory accessible by dd-java-agent at service invocation:

    touch dd-profiler-overrides.jfp
    
  2. Add your desired overrides to the jfp file. For example, to disable allocation profiling and JVM system properties, your dd-profiler-overrides.jfp file would look like the following:

    jdk.ObjectAllocationInNewTLAB#enabled=false
    jdk.ObjectAllocationOutsideTLAB#enabled=false
    jdk.InitialSystemProperty#enabled=false
    
  3. When running your application with dd-java-agent, your service invocation must point to the override file with -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp>, for example:

    java -javaagent:/path/to/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.trace.sample.rate=1 -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp> -jar path/to/your/app.jar
    

Missing profiles in the profile search page

If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

  • Operating system type and version (for example, Linux Ubuntu 20.04)
  • Runtime type, version, and vendor (for example, Python 3.9.5)

Missing profiles in the profile search page

If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

  • Operating system type and version (for example, Linux Ubuntu 20.04)
  • Runtime type, version, and vendor (for example, Go 1.16.5)

Missing profiles in the profile search page

If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

  • Operating system type and version (for example, Linux Ubuntu 20.04)
  • Runtime type, version, and vendor (for example, Ruby 2.7.3)

Application triggers “stack level too deep (SystemStackError)” errors

The profiler instruments the Ruby VM to track thread creation. This instrumentation is compatible with most other Ruby gems that also instrument thread creation, with a few exceptions.

If you’re using any of the below gems:

  • rollbar: Ensure you’re using version 3.1.2 or newer.
  • logging: Disable logging’s thread context inheritance by setting the LOGGING_INHERIT_CONTEXT environment variable to false.

If you’re still experiencing SystemStackError errors after following the above instructions, open a support ticket taking care to include the full backtrace leading to the error.

Missing profiles for Resque jobs

When profiling Resque jobs, you should set the RUN_AT_EXIT_HOOKS environment variable to 1, as described in the Resque documentation.

Without this flag, profiles for short-lived Resque jobs will be unavailable.

Further Reading

Additional helpful documentation, links, and articles: