--- title: Troubleshooting Workload Protection description: >- Troubleshoot Workload Protection, including Agent flares, self tests, and network plugin compatibility. breadcrumbs: >- Docs > Datadog Security > Workload Protection > Troubleshooting Workload Protection --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Troubleshooting Workload Protection If you experience issues with Workload Protection, use the following troubleshooting guidelines. If you need further assistance, contact [Datadog support](https://docs.datadoghq.com/help/). ## Security Agent flare{% #security-agent-flare %} {% alert level="warning" %} From Agent `7.77`, the `security-agent` runtime component for Workload Protection is deprecated and is no longer required. The standalone `security-agent flare` command does not work when the Security Agent process is not running. Use the core Agent `flare` command instead. {% /alert %} Similar to the [Agent flare](https://docs.datadoghq.com/agent/troubleshooting/send_a_flare.md?tab=agent), you can send necessary troubleshooting information to the Datadog support team with one flare command. The flare asks for confirmation before upload, so you may review the content before the Security Agent sends it. In the commands below, replace `` with your Datadog support case ID if you have one, then enter the email address associated with it. If you don't have a case ID, enter the email address you use to log in to Datadog to open a support case. | Platform | Command | | ---------- | --------------------------------------------------------------------------------- | | Docker | `docker exec -it datadog-agent security-agent flare ` | | Kubernetes | `kubectl exec -it -c security-agent -- security-agent flare ` | | Host | `sudo /opt/datadog-agent/embedded/bin/security-agent flare ` | ## Agent self tests{% #agent-self-tests %} To confirm that Workload Protection can detect system events, manually trigger self tests by running the following command: | Platform | Command | | ---------- | ------------------------------------------------------------------------------- | | Docker | `docker exec -it datadog-agent system-probe runtime self-test` | | Kubernetes | `kubectl exec -it -c system-probe -- system-probe runtime self-test` | | Host | `sudo /opt/datadog-agent/embedded/bin/system-probe runtime self-test` | The self-test procedure creates some temporary files and rules to monitor them, and then triggers those rules to confirm that events are propagated correctly. The following response appears when rules are propagated. ``` Runtime self test: OK ``` Events appear in the Events Explorer. ## Compatibility with custom Kubernetes network plugins{% #compatibility-with-custom-kubernetes-network-plugins %} The network based detections of Workload Protection rely on the traffic control sub-system of the Linux kernel. This sub-system is known to introduce race conditions if multiple vendors try to insert, replace, or delete filters on the "clsact" ingress qdisc. Use the following checklist to confirm that Workload Protection is configured correctly: - Check if your vendor uses eBPF traffic control classifiers. If they do not, you can ignore this paragraph. - Check if your vendor returns TC_ACT_OK or TC_ACT_UNSPEC after granting access to a network packet. If they return TC_ACT_UNSPEC, you can ignore this paragraph. - Check which priority your vendor attaches their eBPF classifiers to: - If they use priority 1, Workload Protection network detections do not work inside your containers. - If they use priority 2 to 10, make sure to configure `runtime_security_config.network.classifier_priority` to a number strictly below the priority chosen by your vendor. - If they use priority 11 or higher, you can ignore this paragraph. For example, there is a known race with Cilium 1.9 and lower with the Datadog Agent (version 7.36 to 7.39.1, 7.39.2 excluded) that may happen when a new pod is started. The race can lead to loss of connectivity inside the pod, depending on how Cilium is configured. Ultimately, if the Datadog Agent or your third party vendors cannot be configured to prevent the issue from happening, you should disable the network based detections of Workload Protection by following the steps below: - Add the following parameter to your `system-probe.yaml` configuration file on host based installations: ```yaml runtime_security_config: network: enabled: false ``` - Add the following values if you're using the public Helm Chart to deploy the Datadog Agent: ```yaml datadog: securityAgent: runtime: network: enabled: false ``` - Add the following environment variable if you're deploying the Datadog Agent container manually: ```bash DD_RUNTIME_SECURITY_CONFIG_NETWORK_ENABLED=false ``` ## Troubleshooting Kubernetes remote session or pod admission disruptions{% #troubleshooting-kubernetes-remote-session-or-pod-admission-disruptions %} Workload Protection collects Kubernetes user identities and enriches your Workload Protection events with the context necessary to differentiate remote accesses to your infrastructure from the activity generated by your workloads. This integration relies on a [Kubernetes Mutating Webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to instrument `kubectl exec` sessions. If this instrumentation disrupts the admission of pods or the creation of `kubectl exec` sessions, use the following steps to disable the feature. {% tab title="Datadog Operator" %} 1. Add the following to the `spec` section of the `datadog-agent.yaml` file: ```yaml # datadog-agent.yaml file apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: features: # Integrate with Kubernetes to enrich Workload Protection events with Kubernetes user identities admissionController: cwsInstrumentation: enabled: false ``` 1. Apply the changes and restart the Agent. {% /tab %} {% tab title="Helm" %} 1. Add the following to the `datadog` section of the `datadog-values.yaml` file: ```yaml # datadog-values.yaml file # Integrate with Kubernetes to enrich Workload Protection events with Kubernetes user identities clusterAgent: admissionController: cwsInstrumentation: enabled: false ``` 1. Restart the Agent. {% /tab %} {% tab title="DaemonSet" %} 1. (optional) Add the following setting to the `env` section of `cluster-agent` in the `cluster-agent-deployment.yaml` file: ```bash # Source: datadog/templates/cluster-agent-deployment.yaml apiVersion:app/1 kind: Deployment [...] spec: [...] template: [...] spec: [...] containers: [...] - name: cluster-agent [...] env: - name: DD_RUNTIME_ADMISSION_CONTROLLER_CWS_INSTRUMENTATION_ENABLED value: "false" ``` {% /tab %} ## Disable Workload Protection{% #disable-workload-protection %} To disable Workload Protection, follow the steps for your Agent platform. ### Helm{% #helm %} In the Helm `values.yaml`, set `securityAgent.runtime` to `enabled: false` as follows: In the `values.yaml` file: ```yaml # values.yaml file datadog: # Set to false to Disable CWS securityAgent: runtime: enabled: false ``` ### Daemonset/Docker{% #daemonsetdocker %} Apply the following environment variable change to both the System Probe and the Security Agent deployment for a Daemonset: In the `daemon.json` file: ```json DD_RUNTIME_SECURITY_CONFIG_ENABLED=false ``` ### Host{% #host %} Modify the `system-probe.yaml` and `security-agent.yaml` to disable the runtime config: 1. Disable Workload Protection in `/etc/datadog-agent/system-probe.yaml`. Set `runtime_security_config` to `enabled: false`: In the `system-probe.yaml` file: ```yaml ########################################## ## Security Agent Runtime Configuration ## ## ## ## Settings to send logs to Datadog are ## ## fetched from section `logs_config` ## ## in datadog-agent.yaml ## ########################################## runtime_security_config: ## @param enabled - boolean - optional - default: false ## Set to true to enable full Workload Protection. # enabled: false ## @param fim_enabled - boolean - optional - default: false ## Set to true to only enable the File Integrity Monitoring feature. # fim_enabled: false ## @param socket - string - optional - default: /opt/datadog-agent/run/runtime-security.sock ## The full path of the unix socket where the security runtime module is accessed. # # socket: /opt/datadog-agent/run/runtime-security.sock ``` 1. Disable Workload Protection in `/etc/datadog-agent/security-agent.yaml`. Set `runtime_security_config` to `enabled: false`: In the `security-agent.yaml` file: ```yaml ########################################## ## Security Agent Runtime Configuration ## ## ## ## Settings to send logs to Datadog are ## ## fetched from section `logs_config` ## ## in datadog-agent.yaml ## ########################################## runtime_security_config: ## @param enabled - boolean - optional - default: false ## Set to true to enable the Security Runtime Module. # enabled: false ## @param socket - string - optional - default: /opt/datadog-agent/run/runtime-security.sock ## The full path of the unix socket where the security runtime module is accessed. # # socket: /opt/datadog-agent/run/runtime-security.sock ``` 1. Restart your agents.