Troubleshooting Cloud Security Management Vulnerabilities

Overview

If you experience issues with Cloud Security Management (CSM) Vulnerabilities, use the following troubleshooting guidelines. If you need further assistance, contact Datadog support.

Error messages

Disk space requirements

Ensure your free disk space is equal to the size of your largest container image. This space is needed for the Datadog Agent to scan the container image for vulnerabilities (1 GB by default).

The resulting error appears as:

Error: failed to check current disk usage: not enough disk space to safely collect sbom, 192108482560 available, 1073741824000 required

Workaround:

  • Increase the available disk space to at least 1 GB. If your images are larger than 1 GB, increase your disk space accordingly.
  • If all of your images are smaller than 1 GB, you can decrease the default Agent request disk space with the environment variable: DD_SBOM_CONTAINER_IMAGE_MIN_AVAILABLE_DISK (default value 1GB).

Uncompressed container image layers

The SBOM scan only works with uncompressed container image layers. Certain Kubernetes distributions (such as AWS EKS, minikube, and kind), configure their container runtime to discard the uncompressed layers, causing the scan to fail.

The resulting error appears as:

ERROR | (pkg/workloadmeta/collectors/internal/containerd/image_sbom_trivy.go:80 in func2) | Failed to generate SBOM for containerd image: unable to marshal report to sbom format, err: analyze error: failed to analyze layer:  : unable to get uncompressed layer

The workaround for this issue is to set the configuration option discard_unpacked_layers=false in the containerd configuration file.

  1. Go to Metrics > Summary in Datadog.
  2. Search for the following metrics to aid in troubleshooting:
    • datadog.agent.sbom_attempts: Tracks sbom collection attempts by source and type.
    • datadog.agent.sbom_generation_duration: Measures the time that it takes to generate SBOMs in seconds.
    • datadog.agent.sbom_errors: Number of sbom failures by source, type, and reason.
    • datadog.agent.export_size: The size of the archive written on disk.

Further Reading

Additional helpful documentation, links, and articles:

Setting up container image vulnerabilitiesDOCUMENTATION more more Setting up host vulnerabilitiesDOCUMENTATION more more Enhance your troubleshooting workflow with Container Images in Datadog Container MonitoringBLOG more more