Missing profiles in the profile search page
If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:
- Operating system type and version (for example, Linux Ubuntu 20.04)
- Runtime type, version, and vendor (for example, Ruby 2.7.3)
Application triggers “stack level too deep (SystemStackError)” errors
This issue is not expected to happen since dd-trace-rb version 0.54.0.
If you’re still running into it, open a support ticket taking care to include the full backtrace leading to the error.
Prior to version 0.54.0, the profiler needed to instrument the Ruby VM to track thread creation, which clashed
with similar instrumentation from other gems.
If you’re using any of the below gems:
rollbar: Ensure you’re using version 3.1.2 or newer.logging: Disable logging’s thread context inheritance by setting the LOGGING_INHERIT_CONTEXT environment
variable to false.
Missing profiles for Resque jobs
When profiling Resque jobs, you should set the RUN_AT_EXIT_HOOKS environment
variable to 1, as described in the
Resque documentation.
Without this flag, profiles for short-lived Resque jobs will be unavailable.
There is a known incompatibility between Ruby 2.7 and older GCC versions (4.8 and below) that impacts the profiler (upstream Ruby report, dd-trace-rb bug report). This can result in the following error message: “Your ddtrace installation is missing support for the Continuous Profiler because compilation of the Ruby VM just-in-time header failed. Your C compiler or Ruby VM just-in-time compiler seem to be broken.”
To fix this, update your operating system or Docker image so that the GCC version is something more recent than v4.8.
For further help with this issue, contact support and include the output of running DD_PROFILING_FAIL_INSTALL_IF_MISSING_EXTENSION=true gem install ddtrace and the resulting mkmf.log file.
Frames omitted when backtraces are very deep
The Ruby profiler truncates deep backtraces when collecting profiling data. Truncated backtraces are missing some of their caller functions, making it impossible to link them to the root call frame. As a result, truncated backtraces are grouped together under a N frames omitted frame.
You can increase the maximum depth with the DD_PROFILING_MAX_FRAMES environment variable, or in code:
Datadog.configure do |c|
c.profiling.advanced.max_frames = 500
end
Unexpected run-time failures and errors from Ruby gems that use native extensions in dd-trace-rb 1.11.0+
Starting from dd-trace-rb 1.11.0, the “CPU Profiling 2.0” profiler gathers data by sending SIGPROF unix signals to Ruby applications, enabling finer-grained data gathering.
Sending SIGPROF is a common profiling approach, and may cause system calls from native extensions/libraries to be interrupted with a system EINTR error code.
Rarely, native extensions or libraries called by them may have missing or incorrect error handling for the EINTR error code.
The following incompatibilities are known:
- Using the
mysql2 gem together with versions of libmysqlclient older than 8.0.0. The affected libmysqlclient version is known to be present on Ubuntu 18.04, but not 20.04 or later releases. - Using the
rugged gem.
In these cases, the profiler automatically detects the incompatibility and applies a workaround.
If you encounter run-time failures or errors from Ruby gems that use native extensions, you can revert back to the legacy profiler which does not use SIGPROF signals. To revert to the legacy profiler, set the DD_PROFILING_FORCE_ENABLE_LEGACY environment variable to true, or in code:
Datadog.configure do |c|
c.profiling.advanced.force_enable_legacy_profiler = true
end
Let our team know if you find or suspect any such incompatibilities by opening a support ticket.
Doing this enables Datadog to add them to the auto-detection list, and to work with the gem/library authors to fix the issue.
Further Reading
Additional helpful documentation, links, and articles:
