Overview

Single Step Instrumentation (SSI) helps instrument applications by automatically loading application processes with the Datadog SDKs. SSI works for applications running on Linux hosts, in container environments such as Kubernetes and Docker, and for .NET applications served by Windows IIS—without requiring changes to application dependencies or images. If you encounter issues enabling APM with SSI, use this guide to troubleshoot and resolve common problems. For further assistance, contact Datadog Support.

Troubleshooting methods

You can investigate injection issues in the Datadog UI with Fleet Automation, or manually at the container level. For a lower-level view of what the injector decided for each process, see Injector debug logs.

Troubleshoot injection in Datadog Fleet Automation

Fleet Automation provides two types of instrumentation insights for SSI:

• Process-level insights show instrumentation status and SDK installation details for individual hosts or containers.
• Kubernetes cluster insights provide a higher-level view of instrumentation across your clusters, helping you understand how SSI configuration and injection are applied at scale.

Together, these views let you diagnose injection issues from both the process and cluster perspectives.

Prerequisites

This functionality is available for:

• Languages: Python, Java, Node.js, PHP, .NET
• Environments: Linux hosts, containers, Kubernetes
• Datadog Agent v7.68.2+

View process-level insights

Use process-level insights to verify whether SSI has been correctly applied to your application processes and to identify any injection failures.

1. Navigate to Fleet Automation.
2. Use facets to filter down to relevant hosts:
   • single_step_instrumentation shows which hosts have SSI enabled or disabled.
   • single_step_instrumentation_status shows which hosts have encountered issues with service instrumentation.
3. Select a host to open the Agent details panel.
4. In the Agent panel, go to the Services tab.
5. If SSI is enabled on the host, the tab shows:
   • A banner with the message: “Single Step Instrumentation is enabled on this host.”
   • An SDK Installations section if there are issues to troubleshoot.

View Kubernetes cluster insights

Use cluster-level insights to understand how SSI is configured and functioning across your Kubernetes clusters. These insights extend troubleshooting beyond individual processes to show how instrumentation is applied to workloads at the cluster level.

1. Navigate to Fleet Automation > View Agents, and select Kubernetes Clusters in the upper-right corner.
2. Select a cluster to view its details, including:
   • Whether the cluster is managed by Helm or the Datadog Operator
   • The Cluster Agent and Node Agent versions
   • The integrations and services running on each host
3. Open the Single Step Instrumentation tab to review:
   • The cluster’s SSI configuration (YAML view)
   • The pods identified as instrumentation targets based on cluster configuration or pod-level annotations
   • The status of each targeted pod, including whether instrumentation succeeded
   • The SDKs injected into each pod, including language and version
   • Whether each instrumented workload is generating traces
4. Hover over any status icon to see contextual details about the state of instrumentation or trace collection.

Manually verify injection in the application container

If the Datadog UI does not show any instrumentation issues, or if you’re troubleshooting a single service or container, you can manually verify whether injection occurred as expected. This method is helpful when debugging in environments where centralized visibility is limited or when a specific service isn’t reporting traces.

To confirm injection at the container level, check that:

1. /etc/ld.so.preload includes the following entry:

   /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so
   

2. The LD_PRELOAD environment variable is set to the same value.
3. The directory /opt/datadog-packages/datadog-apm-inject exists, with stable and $version subdirectories.
4. Language-specific directories exist (for example, /opt/datadog/apm/library/java/ for Java).

To enable debug logs during manual verification:

1. Set the following in your pod spec:

      env:
        - name: DD_TRACE_DEBUG    # debug logging for the SDK
          value: "true"
        - name: DD_APM_INSTRUMENTATION_DEBUG    # debug logging for the injector
          value: "true"
      

2. Delete the pod to enable debug logs during injection.

Injector debug logs

Injector debug logs show what the injector decided for each process: whether injection succeeded, was denied, or was skipped, and why. Enable them when Fleet Automation doesn’t explain a failure, when you’re diagnosing at the host or container level, or when you’re collecting information for Datadog Support.

Injector debug logs are separate from tracer debug logs. Use injector logs to diagnose whether and how a tracer was injected; after injection, use tracer debug logs to diagnose the tracer running inside the process.

Enable debug mode

Set the following environment variable on the process you want to instrument:

DD_APM_INSTRUMENTATION_DEBUG=true

This raises the injector log level to DEBUG and adds stderr as a log sink.

The variable must be set in the environment that the injected process inherits:

Host

Export the variable before starting the process:

export DD_APM_INSTRUMENTATION_DEBUG=true
./my-service

Docker

Add the environment variable to the application container:

docker run -e DD_APM_INSTRUMENTATION_DEBUG=true my-image

Debug output appears in the container logs (docker logs <container>).

Kubernetes

Add the variable to the container spec in your pod template:

spec:
  containers:
    - name: my-app
      env:
        - name: DD_APM_INSTRUMENTATION_DEBUG
          value: "true"

Alternatively, add the following pod annotation to enable debug mode without editing the container spec:

metadata:
  annotations:
    admission.datadoghq.com/apm-inject.debug: "true"

Debug output appears in the application pod logs (kubectl logs <pod>).

Review debug logs

Where to find the logs

With debug mode enabled, the injector writes to the stderr of the instrumented process:

Note: To send debug output to a file instead of stderr, set DD_APM_INSTRUMENTATION_OUTPUT_PATHS to an absolute path.

Successful injection

To confirm that injection occurred, look for the log block whose executable matches your runtime. The following example shows a successful Node.js injection:

<DEBUG> ... [linux/process.c:405] process_exe: 'node'
<DEBUG> ... [linux/process.c:443] Main executable path: '/usr/local/bin/node'
<DEBUG> ... [workload_selection.c:147] Workload selection allowed injection: continuing
<DEBUG> ... [workload_selection.c:90] Succesfully loaded policy: 'requirements.bin from SDK policies' from '/opt/datadog-packages/datadog-apm-inject/0.67.0/requirements/nodejs/requirements.bin' [size: 3864]
<DEBUG> ... [languages.c:71] detected language: 'nodejs'
<DEBUG> ... [languages.c:72] detected language version: '20.20.2'
<DEBUG> ... [linux/env_injector.c:38] injection config: DD_TELEMETRY_FORWARDER_PATH=/opt/datadog-packages/datadog-apm-inject/0.67.0/inject/process
<DEBUG> ... [linux/env_injector.c:38] injection config: DD_TAGS=_dd.injection.mode:k8s
<DEBUG> ... [linux/env_injector.c:38] injection config: DD_INJECTION_ENABLED=tracer
<DEBUG> ... [linux/env_injector.c:38] injection config: NODE_OPTIONS=--require /opt/datadog/apm/library/js/node_modules/dd-trace/init.js
<INFO>  ... [./libinject.c:81] injection duration: 1.066500 ms
<DEBUG> ... [injection_metadata.c:101] sending injection-metadata telemetry: result='0', result_reason='injection completed successfully'
<DEBUG> ... [./libinject.c:231] injector finished

Common log messages

Injection disabled

If DD_INSTRUMENT_SERVICE_WITH_APM=false or injection is otherwise disabled:

<DEBUG> ... [libinject.c:115] disabled flag set, not injecting

The injector loaded but intentionally skipped injection. Remove DD_INSTRUMENT_SERVICE_WITH_APM=false or set it to true to allow injection.

No runtime detected

If the process is not a supported language runtime:

<DEBUG> ... [libinject.c:175] No known runtime was detected - not injecting!

The injector ran but the process is not one it instruments (nodejs, java, python, ruby, dotnet, php). This is expected for non-application processes.

Workload selection denied

<DEBUG> ... [workload_selection.c:149] Workload selection denied injection

A policy prevented injection for this process. The preceding Evaluating '<policy-name>' lines show which rule matched and the values it compared.

Re-exec detected

<DEBUG> ... [libinject.c:102] Re-exec detected!

The process re-executed itself so that the injected environment variables take effect. This is expected behavior. Injection continues in the re-executed process, which produces its own set of debug lines.

Configuration options that affect injection

There are several configuration mechanisms that can block or alter injection behavior.

Storage requirements

SSI downloads language SDKs and an injector package onto each host. The amount of disk space required depends on the number of languages in use and the number of pods being instrumented. A rough estimate is:

[sum of the language library sizes]
+
[injector package size] * [number of injected pods per host]

Because library packages are updated frequently and may grow when support for new language versions is added, disk usage can change over time. If your environment has limited disk space, monitor package sizes and allow extra capacity to avoid injection failures.

Injector version

To set the injector version:

• At the cluster level:

  Set in values.yaml under datadog.apm.instrumentation.injector.imageTag.

• At the pod level:

  Set with the annotation admission.datadoghq.com/apm-inject.version.

For host or Docker injection, modifying the auto_inject version is not recommended.

Allow and deny lists

Default deny list

Datadog maintains an internal deny list to prevent injection into certain processes (for example, IDEs or databases). If a process command or entrypoint is on this list, the injector skips the injection process.

Linux instrumentation rules

Kubernetes instrumentation rules

Instrumentation rules enable injection based on Kubernetes labels and selectors. Rules to consider:

1. disabledNamespaces always takes precedence.
2. When a pod initializes, the target list is checked from top to bottom. Only the first matching rule applies per pod.

Injection container flagged by security scanners

Security tools may flag the apm-inject container because it runs an executable at startup, which can resemble malicious software.

The container’s behavior is expected and safe; the executable configures the environment for auto-instrumentation.

Datadog adheres to security best practices and is working with security vendors to allowlist this container.

Environments with strict pod security settings

If pod security rules block the Datadog init container, you may see errors like:

Privilege escalation container is not allowed or violates PodSecurity "restricted: latest": allowPrivilegeEscalation is false

To resolve this, set one of the following Cluster Agent options:

• DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_INIT_SECURITY_CONTEXT
• admission_controller.auto_instrumentation.init_security_context

The value should be a JSON string that applies the necessary security context to the Datadog init containers.

Custom instrumentation

Custom instrumentation still requires you to import the SDK. Configuration variables like .NET’s DD_TRACE_METHODS remain available for defining custom spans.

General troubleshooting

SSI continues running after setting DD_TRACE_ENABLED=false

Setting DD_TRACE_ENABLED=false does not prevent SSI from loading the SDK. The injector runs before the SDK evaluates its environment variables, so SDK-level environment variables have no effect on SSI. To disable or remove SSI, see your platform’s SSI setup page.

Environment-specific troubleshooting

Host and Docker environments

Host injection does not apply to existing processes

The preload library only injects into newly launched processes. Start a new shell session or log out and log back in to apply instrumentation.

Note: Docker-based injection does not have this limitation.

Injection fails on small instance types

The preload library allows the analyzer one second to complete its work. On small VM instances running multiple services (for example, t2.micro), this time limit may be exceeded. To overcome this issue, use a larger instance size, such as t2.small.

Errors after manual uninstallation of agent files

If you manually delete agent files, you may see errors like:

ERROR: ld.so: object /opt/datadog/apm/inject/launcher.preload.so from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored

To properly uninstall SSI, follow the platform-specific instructions:

• Kubernetes
• Docker
• Linux
• Windows

Injection not working with rootless Docker

When using rootless Docker, set docker_socket in /etc/datadog-agent/inject/docker_config.yaml to the path of the Docker socket used by the current user (typically /run/user/$UID/docker.sock). No reboot is required.

Injection fails with statically linked launchers

If a custom launcher is statically linked (common with Go), the preload library might not be invoked. Injection can still succeed if:

• The launcher’s command line includes the language name
• The launcher runs an intermediary dynamically linked program

However, direct process launches from statically linked binaries are not injected.

Kubernetes environments

The Datadog Admission Controller must be deployed and configured before application pods are created; it cannot modify existing pods.

To troubleshoot Admission Controller issues:

1. Check Cluster Agent pod health:

   kubectl get pods
   kubectl get deployments
   

2. Check the Cluster Agent leader logs for INFO messages indicating successful Admission Controller startup. For example:

   Group version 'admissionregistration.k8s.io/v1' is available, Starting secrets controller, Starting webhook controller
   

3. Check the Admission Controller status by doing one of the following:

   • Run agent status inside the Cluster Agent pod to get a live status output.
   • If troubleshooting retrospectively, check status.log within a flare. When a flare is generated, the system runs agent status and stores the output in status.log.

   In both cases, find the Admission Controller and Webhooks sections, and verify the following:

   • All expected MutatingWebhookConfiguration resources are listed (for auto-instrumentation, configuration injection, and tag injection).
   • The webhook configurations reference the correct Secret.
   • The CA bundle digest matches across configurations.

4. Inspect injection attempts in telemetry.log or in the output of the following command:

   kubectl exec -it <cluster agent pod> agent telemetry
   

   Look for admission_webhooks_library_injection_attempts to see injection attempts by language.

Failed mutations

The Cluster Agent logs warnings and errors for injection failures, typically from admission/server.go. For example, a warning might appear if JAVA_TOOL_OPTIONS is set using valueFrom.

Use the metric datadog.cluster_agent.admission_webhooks.library_injection_errors for further debugging.

Language annotation cannot be applied

During setup, SSI detects the application language of your service and applies a service label in the form internal.dd.datadoghq.com/service-name.detected_langs. If the label cannot be applied, injection fails.

Sometimes, labeling errors occur because a service name breaks Kubernetes string limits (63 characters). For example:

languagedetection/patcher.go:231 in handleDeploymentEvent) | failed to handle deployment event: annotations: Invalid value: "internal.dd.datadoghq.com/dummy-python-container-long-long-long-long-long-x.detected_langs": name part must be no more than 63 characters

String limit violations are common if service tags are not explicitly set through Unified Service Tagging, in which case default image names are used.

Injection appears successful but traces are missing

If logs show no issues but traces are missing, there may be an application-side misconfiguration. Verify that:

• Required annotations and labels are present.
• Unified Service Tagging is set up correctly.
• Allow/deny lists for instrumentation rules are properly defined.

Language-specific troubleshooting

Java

JAVA_TOOL_OPTIONS is too long

The JAVA_TOOL_OPTIONS environment variable has a JVM-enforced limit of 1024 characters. During injection, Datadog appends a -javaagent flag to this variable to enable tracing. If the combined value exceeds the limit, the JVM emits a warning and ignores the variable, preventing injection.

To avoid this issue, exclude the affected process from injection.

JAVA_TOOL_OPTIONS changes program output

When JAVA_TOOL_OPTIONS is set, the JVM prints a message to stdout, such as Picked up JAVA_TOOL_OPTIONS: -Xmx1024m. If a process reads and depends on this output, it may be affected.

As of version 0.12.2, injection is skipped for java -version to avoid interfering with processes that parse its output.

Multiple Java sites report under the same service name

By default, Single Step sets the DD_SERVICE environment variable, which applies a single service name across all web applications running on the same server (such as Tomcat or WebLogic). As a result, all sites report under the same name.

Use one of the following options to enable split-by-tags so that each site reports under its own name:

• JVM system property: -Ddd.trace.split-by-tags=servlet.context
• Environment variable: DD_TRACE_SPLIT_BY_TAGS=servlet.context

Tracer already exists

SSI does not inject into applications that already use a -javaagent option or other tracing configuration.

Ruby

Ruby injection modifies the Gemfile to add the Datadog SDK. If injection support is later removed, the application may fail to start due to the missing dependency.

To resolve this, restore the original Gemfile. If you still want to use APM after removing injection, run bundle install to download the gem.

Python

Versions <=2.7.5 contain a pre-packaged protobuf dependency that can conflict with system libraries.

.NET

SSI is applied but no .NET traces reach the Agent

If SSI annotations and init containers are present on the pod but no .NET traces arrive, another profiler may have precedence. Check CORECLR_PROFILER on the main container. If the value is not {846F5F1C-F9AE-4B07-969E-05C26BC060D8} (the Datadog .NET tracer CLSID), another profiler is loaded instead of Datadog.

Remove the conflicting CORECLR_* environment variables (and any LD_PRELOAD entries that reference the other profiler) from the source that injected them: another vendor’s operator, init container, pod template, or Helm values. Then roll the pods. The .NET CLR Profiling API allows only one subscriber per process.

Collect diagnostic information for support

When contacting support about injection issues, collect the following information to assist troubleshooting:

1. Are you using host injection, Docker injection, or both?

2. Verify that the /opt/datadog-packages/datadog-apm* directories exist.

3. For host injection, check for the existence and permissions of /etc/ld.so.preload:

   sudo ls -l /etc/ld.so.preload
   

   It should be owned by root with 644 permissions (-rw-r--r--).

4. Enable injector debug logs and collect the output. For instructions, see Injector debug logs.

5. Provide an Agent flare.

Additional information for Kubernetes-based injection

Collect the following details if troubleshooting injection in a Kubernetes environment:

• The method used to deploy the Cluster Agent (for example, Helm, Datadog Operator, or kubectl commands).
• Deployment files for the application pod.
• Flares from both the Node Agent and the Cluster Agent, ideally with DEBUG mode enabled.
• Output of:

  kubectl describe pod <app pod>
  

• Injector debug logs from the application pod (not the Cluster Agent). For instructions, see Injector debug logs.

Further reading

役立つドキュメント、リンク、記事:

Single Step APM InstrumentationDOCUMENTATION Troubleshooting APM Instrumentation on a HostLEARNING CENTER