--- title: Troubleshooting the .NET Profiler description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Continuous Profiler > Profiler Troubleshooting > Troubleshooting the .NET Profiler --- # Troubleshooting the .NET Profiler ## Missing profiles in the profile search page{% #missing-profiles-in-the-profile-search-page %} If you've configured the profiler and don't see profiles in the profile search page, here are things to check, depending on the operating system. {% tab title="Linux with Single Step APM Instrumentation" %} 1. Check that the Agent is installed and running. 1. Check that the profiler has been loaded from the loader log: 1. Open the `dotnet-native-loader-dotnet-` log file in the `/var/log/datadog` folder. 1. Look for `CorProfiler::Initialize: Continuous Profiler initialized successfully.` near the end. If that message is not present, enable debug logs by setting the `DD_TRACE_DEBUG` environment variable for the application. 1. Restart the application. 1. Open the `dotnet-native-loader-dotnet-` log file in the `/var/log/datadog` folder. 1. Look for the `#Profiler` entry. 1. Check the following lines to ensure that the profiler library has been loaded: ```gdscript3 [...] #Profiler [...] PROFILER;{BD1A650D-AC5D-4896-B64F-D6FA25D6B26A};win-x64;.\Datadog.Profiler.Native.dll [...] PROFILER;{BD1A650D-AC5D-4896-B64F-D6FA25D6B26A};win-x86;.\Datadog.Profiler.Native.dll [...] DynamicDispatcherImpl::LoadConfiguration: [PROFILER] Loading: .\Datadog.Profiler.Native.dll [AbsolutePath=/opt/datadog/linux-x64/./Datadog.Tracer.Native.so] [...] [PROFILER] Creating a new DynamicInstance object [...] Load: /opt/datadog/linux-x64/./Datadog.Tracer.Native.so [...] GetFunction: DllGetClassObject [...] GetFunction: DllCanUnloadNow ``` 1. Check the result of profiles export: 1. If debug logs were not enabled in step 2.2, set the `DD_TRACE_DEBUG` environment variable to `true` for the application and restart it. 1. Open the `DD-DotNet-Profiler-Native--` log file in the `/var/log/datadog` folder. 1. Look for `libddprof error: Failed to send profile.` entries: this message means it can't contact the agent. Ensure the `DD_TRACE_AGENT_URL` is set to the correct Agent URL. See [Enabling the .NET Profiler-Configuration](https://docs.datadoghq.com/profiler/enabling/dotnet.md?tab=linux#configuration) for more information. 1. If the `Failed to send profile` message is not present, look for `The profile was sent. Success?` entries. The following message means the profile has been sent successfully: ``` true, Http code: 200 ``` 1. Check the other HTTP codes for possible errors such as 403 for invalid API key. 1. For missing CPU or Wall time profiles only, check that the Datadog signal handler for stack walk has not been replaced: 1. Open the `DD-DotNet-Profiler-Native--` log file in the `/var/log/datadog` folder. 1. Look for these two messages: - `Profiler signal handler was replaced again. It will not be restored: the profiler is disabled.` - `Fail to restore profiler signal handler.` 1. If one of these messages is present, it means that the application code or a third party code is repeatedly reinstalling its own signal handler over the Datadog signal handler. To avoid any further conflict, the CPU and Wall time profilers are disabled. Note that the following message could appear, but it does not impact Datadog profiling: `Profiler signal handler has been replaced. Restoring it.` This indicates only that the Datadog signal handler is reinstalled when it was overwritten. 1. Confirm that the following environment variables are not set: - `CORECLR_ENABLE_PROFILING` - `CORECLR_PROFILER` - `CORECLR_PROFILER_PATH` - The value pointing to `Datadog.Linux.ApiWrapper.x64.so` in `LD_PRELOAD` If these variables have been set, remove them from your scripts or Dockerfile. Otherwise, a previous manually-installed version of the profiler is used. 1. Check that the value of `DD_PROFILING_ENABLED` is set to `1` or `Auto`. If set to `Auto`, the profiler generates profiles only if both of the following conditions are met: - A trace was created - The application ran for more than 30 seconds If it is set to another value or not set at all, the profiler is disabled. {% /tab %} {% tab title="Linux" %} 1. Check that the Agent is installed and running. 1. Check that the profiler has been loaded from the loader log: 1. Open the `dotnet-native-loader-dotnet-` log file in the `/var/log/datadog` folder. 1. Look for `CorProfiler::Initialize: Continuous Profiler initialized successfully.` near the end. If that message is not present, enable debug logs by setting the `DD_TRACE_DEBUG` environment variable for the application. 1. Restart the application. 1. Open the `dotnet-native-loader-dotnet-` log file in the `/var/log/datadog` folder. 1. Look for the `#Profiler` entry. 1. Check the following lines to ensure that the profiler library has been loaded: ```gdscript3 [...] #Profiler [...] PROFILER;{BD1A650D-AC5D-4896-B64F-D6FA25D6B26A};win-x64;.\Datadog.Profiler.Native.dll [...] PROFILER;{BD1A650D-AC5D-4896-B64F-D6FA25D6B26A};win-x86;.\Datadog.Profiler.Native.dll [...] DynamicDispatcherImpl::LoadConfiguration: [PROFILER] Loading: .\Datadog.Profiler.Native.dll [AbsolutePath=/opt/datadog/linux-x64/./Datadog.Tracer.Native.so] [...] [PROFILER] Creating a new DynamicInstance object [...] Load: /opt/datadog/linux-x64/./Datadog.Tracer.Native.so [...] GetFunction: DllGetClassObject [...] GetFunction: DllCanUnloadNow ``` 1. Check that the value of `DD_PROFILING_ENABLED` is set to `1`; if it is not set or with another value, the profiler is disabled. 1. Check the result of profiles export: 1. If debug logs were not enabled in step 2.b, set the `DD_TRACE_DEBUG` environment variable to `true` for the application and restart it. 1. Open the `DD-DotNet-Profiler-Native--` log file in the `/var/log/datadog` folder. 1. Look for `libddprof error: Failed to send profile.` entries: this message means it can't contact the agent. Ensure the `DD_TRACE_AGENT_URL` is set to the correct Agent URL. See [Enabling the .NET Profiler-Configuration](https://docs.datadoghq.com/profiler/enabling/dotnet.md?tab=linux#configuration) for more information. 1. If the `Failed to send profile` message is not present, look for `The profile was sent. Success?` entries. The following message means the profile has been sent successfully: ``` true, Http code: 200 ``` 1. Check the other HTTP codes for possible errors such as 403 for invalid API key. 1. For missing CPU or Wall time profiles only, check that the Datadog signal handler for stack walk has not been replaced: 1. Open the `DD-DotNet-Profiler-Native--` log file in the `/var/log/datadog` folder. 1. Look for these two messages: - `Profiler signal handler was replaced again. It will not be restored: the profiler is disabled.` - `Fail to restore profiler signal handler.` 1. If one of these messages is present, it means that the application code or a third party code is repeatedly reinstalling its own signal handler over the Datadog signal handler. To avoid any further conflict, the CPU and Wall time profilers are disabled. Note that the following message could appear, but it does not impact Datadog profiling: `Profiler signal handler has been replaced. Restoring it.` This indicates only that the Datadog signal handler is reinstalled when it was overwritten. {% /tab %} {% tab title="Windows" %} The default profiler log directory is `%ProgramData%\Datadog .NET Tracer\logs\`. Prior to v2.24, the default directory was `%ProgramData%\Datadog-APM\logs\DotNet`. 1. Check that the Agent is installed and running and is visible in the Windows Services panel. 1. Check that the profiler has been loaded from the loader log: 1. Open the `dotnet-native-loader--` log file from the default log folder. 1. Look for `CorProfiler::Initialize: Continuous Profiler initialized successfully.` near the end. If the `initialized successfully` message is not present, enable debug logs by setting the `DD_TRACE_DEBUG` environment variable for the application. 1. Restart the application. 1. Open the `dotnet-native-loader--` log file from the default log folder. 1. Look for the `#Profiler` entry. 1. Check the following lines to ensure that the profiler library has been loaded: ```gdscript3 [...] #Profiler [...] PROFILER;{BD1A650D-AC5D-4896-B64F-D6FA25D6B26A};win-x64;.\Datadog.Profiler.Native.dll [...] PROFILER;{BD1A650D-AC5D-4896-B64F-D6FA25D6B26A};win-x86;.\Datadog.Profiler.Native.dll [...] DynamicDispatcherImpl::LoadConfiguration: [PROFILER] Loading: .\Datadog.Profiler.Native.dll [AbsolutePath=C:\Program Files\Datadog\.NET Tracer\win-x64\Datadog.Profiler.Native.dll] [...] [PROFILER] Creating a new DynamicInstance object [...] Load: C:\Program Files\Datadog\.NET Tracer\win-x64\Datadog.Profiler.Native.dll [...] GetFunction: DllGetClassObject [...] GetFunction: DllCanUnloadNow ``` 1. Check that the value of `DD_PROFILING_ENABLED` is set to `1`; if it is not set, or set with another value, the profiler is disabled. 1. Check the result of profiles export: 1. If debug logs were not enabled in step 2.b, set the `DD_TRACE_DEBUG` environment variable to `true` for the application and restart it. 1. Open the `DD-DotNet-Profiler-Native--` log file from the default log folder. 1. Look for `libddprof error: Failed to send profile.` entries: This message means that it can't contact the agent. Ensure the `DD_TRACE_AGENT_URL` is set to the correct Agent URL. See [Enabling the .NET Profiler-Configuration](https://docs.datadoghq.com/profiler/enabling/dotnet.md?tab=linux#configuration) for more information. 1. If the `Failed to send profile` message is not present, look for `The profile was sent. Success?` entries. The following message means the profile has been sent successfully: ``` true, Http code: 200 ``` 1. Check the other HTTP codes for possible errors such as 403 for invalid API key. {% /tab %} Otherwise, turn on [debug mode](https://docs.datadoghq.com/tracing/troubleshooting.md#debugging-and-logging) and [open a support ticket](https://docs.datadoghq.com/help/) with the debug files and the following information: - Operating system type and version (for example, Windows Server 2019 or Ubuntu 20.04). - Runtime type and version (for example, .NET Framework 4.8 or .NET Core 6.0). - Application type (for example, Web application running in IIS). ## Reduce overhead when using the profiler{% #reduce-overhead-when-using-the-profiler %} The [different profile types](https://docs.datadoghq.com/profiler/profile_types.md?code-lang=dotnet) have a fixed CPU and memory overhead, so *the more profiled applications, the higher the overhead.* ### Avoid enabling the profiler machine-wide{% #avoid-enabling-the-profiler-machine-wide %} Datadog does not recommend enabling the profiler at machine-level or for all IIS application pools. To reduce the amount of resources used by the profiler, you can: - Increase the allocated resources, such as increasing CPU cores. - Profile only specific applications by setting environment in batch files instead of directly running the application. - Reduce the number of IIS pools being profiled (only possible in IIS 10 or later). - Disable wall time profiling with the setting `DD_PROFILING_WALLTIME_ENABLED=0`. ### Linux Containers{% #linux-containers %} The exact value can vary but the fixed overhead cost means that the relative overhead of the profiler can be significant in very small containers. To avoid this situation, the profiler is disabled in containers with less than one core. You can override the one core threshold by setting the `DD_PROFILING_MIN_CORES_THRESHOLD` environment variable to a value less than one. For example, a value of `0.5` allows the profiler to run in a container with at least 0.5 cores. However, in that case, there will be a CPU consumption increase, even for idle services, because the profiler threads always scan the application's threads. The less available core, the more the CPU consumption increases. Disabling the wall time profiler with the setting `DD_PROFILING_WALLTIME_ENABLED=0` decreases the CPU consumption by the profiler. If this is not enough, increase the CPU cores available for your containers. ### Disabling the profiler{% #disabling-the-profiler %} Since APM tracing also relies on the CLR Profiling API, if you want to stop collecting .NET profiles but continue receiving .NET traces, set the following environment variables to disable only profiling. ``` DD_PROFILING_ENABLED=0 CORECLR_ENABLE_PROFILING=1 ``` ## No CPU or Wall time because the application on Linux is hung{% #no-cpu-or-wall-time-because-the-application-on-linux-is-hung %} If an application hangs, or otherwise becomes unresponsive on Linux, CPU and Wall time samples are no longer available, follow these steps: 1. Open the `DD-DotNet-Profiler-Native--` log file in the `/var/log/datadog/dotnet` folder. 1. Look for `StackSamplerLoopManager::WatcherLoopIteration - Deadlock intervention still in progress for thread ...`. If this message is not present, the rest does not apply. 1. If the message is found, it means that the stack walking mechanism could be deadlocked. To investigate the issue, dump the call stacks of all threads in the application. For example, to do this with the gdb debugger: 1. Install gdb. 1. Run the following command: ``` gdb -p -batch -ex "thread apply all bt full" -ex "detach" -ex "quit" ``` 1. Send the resulting output to [Datadog Support](https://docs.datadoghq.com/help/). ## Further Reading{% #further-reading %} - [APM Troubleshooting](https://docs.datadoghq.com/tracing/troubleshooting.md)