Injecting Libraries Locally

이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Overview

To automatically instrument your application, you can:

For more information, see Automatic Instrumentation.

How to inject the library locally, without touching the application code at all, varies depending on where and how your Agent and application are installed. Select the scenario that represents your environment:

With the Admission Controller approach, the Agent uses the Kubernetes Admission Controller to intercept requests to the Kubernetes API and mutate new pods to inject the specified instrumentation library.

Library injection is applied on new pods only and does not have any impact on running pods.

To learn more about Kubernetes Admission Controller, read Kubernetes Admission Controllers Reference.

Requirements

  • Kubernetes v1.14+
  • Datadog Cluster Agent v7.40+ for Java, Python, Node.js, Datadog Cluster Agent v7.44+ for .NET and Ruby.
  • Datadog Admission Controller enabled. Note: In Helm chart v2.35.0 and later, Datadog Admission Controller is activated by default in the Cluster Agent.
  • For Python, uWSGI applications are not supported at this time.
  • For Ruby, library injection support is in Beta. Instrumentation is only supported for Ruby on Rails applications with Bundler version greater than 2.3 and without vendored gems (deployment mode or BUNDLE_PATH).
  • Applications in Java, JavaScript, Python, .NET, or Ruby deployed on Linux with a supported architecture. Check the corresponding container registry for the complete list of supported architectures by language.

Container registries

Docker Hub is subject to image pull rate limits. If you are not a Docker Hub customer, Datadog recommends that you update your Datadog Agent and Cluster Agent configuration to pull from GCR or ECR. For instructions, see Changing your container registry.

Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and Amazon ECR:

The DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is gcr.io/datadoghq.

You can pull the tracing library from a different registry by changing it to docker.io/datadog, public.ecr.aws/datadog, or another URL if you are hosting the images in a local container registry.

Configure instrumentation libraries injection

For your Kubernetes applications whose traces you want to send to Datadog, configure the Datadog Admission Controller to inject Java, JavaScript, Python, .NET or Ruby instrumentation libraries automatically. From a high level, this involves the following steps, described in detail below:

  1. Enable Datadog Admission Controller to mutate your pods.
  2. Annotate your pods to select which instrumentation library to inject.
  3. Tag your pods with Unified Service Tags to tie Datadog telemetry together and navigate seamlessly across traces, metrics, and logs with consistent tags.
  4. Apply your new configuration.
You do not need to generate a new application image to inject the library. The library injection is taken care of adding the instrumentation library, so no change is required in your application image.

Step 1 - Enable Datadog Admission Controller to mutate your pods

By default, Datadog Admission controller mutates only pods labeled with a specific label. To enable mutation on your pods, add the label admission.datadoghq.com/enabled: "true" to your pod spec.

Note: You can configure Datadog Admission Controller to enable injection config without having this pod label by configuring the Cluster Agent with clusterAgent.admissionController.mutateUnlabelled (or DD_ADMISSION_CONTROLLER_MUTATE_UNLABELLED) to true.

For more details on how to configure, read Datadog Admission Controller page.

For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    # (...)
spec:
  template:
    metadata:
      labels:
        admission.datadoghq.com/enabled: "true" # Enable Admission Controller to mutate new pods part of this deployment
    spec:
      containers:
        - # (...)

Step 2 - Annotate your pods for library injection

To select your pods for library injection, use the annotations provided in the following table within your pod spec:

LanguagePod annotation
Javaadmission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
JavaScriptadmission.datadoghq.com/js-lib.version: "<CONTAINER IMAGE TAG>"
Pythonadmission.datadoghq.com/python-lib.version: "<CONTAINER IMAGE TAG>"
.NETadmission.datadoghq.com/dotnet-lib.version: "<CONTAINER IMAGE TAG>"
Rubyadmission.datadoghq.com/ruby-lib.version: "<CONTAINER IMAGE TAG>"

The available library versions are listed in each container registry, as well as in the tracer source repositories for each language:

  • Java
  • JavaScript
  • Python
  • .NET
    • Note: For .NET library injection on musl-based Linux distributions (such as Alpine), append -musl to the pod annotation tag for versions 2.55 and earlier. For example, use v2.29.0-musl instead of v2.29.0. The -musl suffix is not required for versions after 2.55, including v3.0+.
  • Ruby

Note: If you already have an application instrumented using version X of the library, and then use library injection to instrument using version Y of the same tracer library, the tracer does not break. Rather, the library version loaded first is used. Because library injection happens at the admission controller level prior to runtime, it takes precedence over manually configured libraries.

Note: Using the latest tag is supported, but use it with caution because major library releases can introduce breaking changes.

For example, to inject a Java library:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    # (...)
spec:
  template:
    metadata:
      labels:
        admission.datadoghq.com/enabled: "true" # Enable Admission Controller to mutate new pods in this deployment
      annotations:
        admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
    spec:
      containers:
        - # (...)

Step 3 - Tag your pods with Unified Service Tags

With Unified Service Tags, you can tie Datadog telemetry together and navigate seamlessly across traces, metrics, and logs with consistent tags. Set the Unified Service Tagging on both the deployment object and the pod template specs. Set Unified Service tags by using the following labels:

  metadata:
    labels:
      tags.datadoghq.com/env: "<ENV>"
      tags.datadoghq.com/service: "<SERVICE>"
      tags.datadoghq.com/version: "<VERSION>"

Note: It is not necessary to set the environment variables for universal service tagging (DD_ENV, DD_SERVICE, DD_VERSION) in the pod template spec, because the Admission Controller propagates the tag values as environment variables when injecting the library.

For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    tags.datadoghq.com/env: "prod" # Unified service tag - Deployment Env tag
    tags.datadoghq.com/service: "my-service" # Unified service tag - Deployment Service tag
    tags.datadoghq.com/version: "1.1" # Unified service tag - Deployment Version tag
  # (...)
spec:
  template:
    metadata:
      labels:
        tags.datadoghq.com/env: "prod" # Unified service tag - Pod Env tag
        tags.datadoghq.com/service: "my-service" # Unified service tag - Pod Service tag
        tags.datadoghq.com/version: "1.1" # Unified service tag - Pod Version tag
        admission.datadoghq.com/enabled: "true" # Enable Admission Controller to mutate new pods part of this deployment
      annotations:
        admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
    spec:
      containers:
        - # (...)

Step 4 - Apply the configuration

Your pods are ready to be instrumented when their new configuration is applied.

The library is injected on new pods only and does not have any impact on running pods.

Check that the library injection was successful

Library injection leverages the injection of a dedicated init container in pods. If the injection was successful you can see an init container called datadog-lib-init in your pod:

Kubernetes environment details page showing init container in the pod.

Or run kubectl describe pod <my-pod> to see the datadog-lib-init init container listed.

The instrumentation also starts sending telemetry to Datadog (for example, traces to APM).

Troubleshooting installation issues

If the application pod fails to start, run kubectl logs <my-pod> --all-containers to print out the logs and compare them to the known issues below.

.NET installation issues

dotnet: error while loading shared libraries: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
  • Problem: The pod annotation for the dotnet library version included a -musl suffix, but the application container runs on a Linux distribution that uses glibc.
  • Solution: Remove the -musl suffix from the dotnet library version.
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /datadog-lib/continuousprofiler/Datadog.Linux.ApiWrapper.x64.so)
  • Problem: The application container runs on a Linux distribution that uses musl-libc (for example, Alpine), but the pod annotation does not include the -musl suffix.
  • Solution: Add the -musl suffix to the dotnet library version.

Python installation issues

Noisy library logs

In Python < 1.20.3, Python injection logs output to stderr. Upgrade to 1.20.3 or above to suppress the logs by default. The logs can be enabled by setting the environment variable DD_TRACE_DEBUG to 1.

Incompatible Python version

The library injection mechanism for Python only supports injecting the Python library in Python v3.7+.

user-installed ddtrace found, aborting
  • Problem: The ddtrace library is already installed on the system so the injection logic aborts injecting the library to avoid introducing a breaking change in the application.
  • Solution: Remove the installation of ddtrace if library injection is desired. Otherwise, use the installed library (see documentation) instead of library injection.

TLS issues

tls: protocol version not supported
  • Problem: Since Cluster Agent v1.20, the API is only served using TLS v1.3 by default. If the Kubernetes cluster is configured with TLS v1.2 or older, library injection fails.

  • Solution: Set DD_CLUSTER_AGENT_ALLOW_LEGACY_TLS to true for Cluster Agent.

Tracing Library Injection on a host is in beta.

When both the Agent and your services are running on a host, real or virtual, Datadog injects the tracing library by using a preload library that overrides calls to execve. Any newly started processes are intercepted and the specified instrumentation library is injected into the services.

Note: Injection on arm64 is not supported.

Install both library injection and the Datadog Agent

Requirements: A host running Linux.

If the host does not yet have a Datadog Agent installed, or if you want to upgrade your Datadog Agent installation, use the Datadog Agent install script to install both the injection libraries and the Datadog Agent:

DD_APM_INSTRUMENTATION_ENABLED=host DD_APM_INSTRUMENTATION_LIBRARIES=java:1,python:2,js:5,dotnet:3,ruby:2 DD_API_KEY=<YOUR KEY> DD_SITE="<YOUR SITE>" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

By default, running the script installs support for Java, Node.js, Python, Ruby, and .NET all pinned to the latest major version. If you want to specify which language support is installed, also set the DD_APM_INSTRUMENTATION_LIBRARIES environment variable. The valid values are java, js, python, ruby, and dotnet. Use a comma-separated list to specify more than one language:

DD_APM_INSTRUMENTATION_LIBRARIES=java:1,js:5 DD_APM_INSTRUMENTATION_ENABLED=host DD_API_KEY=<YOUR KEY> DD_SITE="<YOUR SITE>" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

Exit and open a new shell to use the injection library.

Next steps

If you haven’t already, install your app and any supporting languages or libraries it requires.

When an app that is written in a supported language is launched, it is automatically injected with tracing enabled.

Configure the injection

Configure host injection in one of the following ways:

  • Set environment variables on the process being launched.

Values in environment variables override settings in the configuration file on a per-process basis.

Configuration file

Property namePurposeDefault valueValid values
log_levelThe logging leveloffoff, debug, info, warn, error
output_pathsThe location where log output is writtenstderrstderr or a file:// URL
envThe default environment assigned to a processnonen/a

Example

---
log_level: debug
output_paths:
  - file:///tmp/host_injection.log
env: dev

Environment variables

The following environment variables configure library injection. You can pass these in by export through the command line (export DD_CONFIG_SOURCES=BASIC), shell configuration, or launch command.

Each of the fields in the config file corresponds to an environment variable. This environment variable is read from the environment of the process that’s being launched and affects only the process currently being launched.

Config file propertyEnvironment Variable
log_levelDD_APM_INSTRUMENTATION_DEBUG
output_pathsDD_APM_INSTRUMENTATION_OUTPUT_PATHS
envDD_ENV

The DD_APM_INSTRUMENTATION_DEBUG environment variable is limited to the values true and false (default value false). Setting it to true sets log_level to debug and setting it to false (or not setting it at all) uses the log_level specified in the configuration file. The environment variable can only set the log level to debug, not any other log level values.

The DD_INSTRUMENT_SERVICE_WITH_APM environment variable controls whether or not injection is enabled. It defaults to TRUE. Set it to FALSE to turn off library injection altogether.

Default configuration

By default, the following settings are enabled in an instrumented process:

  • Tracing
  • Log injection, assuming the application uses structured logging (usually JSON). For traces to appear in non-structured logs, you must change your application’s log configuration to include placeholders for trace ID and span ID. See Connect Logs and Traces for more information.
  • Health metrics
  • Runtime metrics

Exercise your application to start generating telemetry data, which you can see as traces in APM.

Tracing Library Injection on hosts and containers is in beta.

When your Agent is running on a host, and your services are running in containers, Datadog injects the tracing library by intercepting container creation and configuring the Docker container.

Any newly started processes are intercepted and the specified instrumentation library is injected into the services.

Note: Injection on arm64 is not supported.

Install both library injection and the Datadog Agent

Requirements:

If the host does not yet have a Datadog Agent installed, or if you want to upgrade your Datadog Agent installation, use the Datadog Agent install script to install both the injection libraries and the Datadog Agent:

DD_APM_INSTRUMENTATION_ENABLED=all DD_API_KEY=<YOUR KEY> DD_SITE="<YOUR SITE>" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

By default, running the script installs support for Java, Node.js, Python, Ruby, and .NET. If you want to specify which language support is installed, also set the DD_APM_INSTRUMENTATION_LIBRARIES environment variable. The valid values are java, js, python, ruby, and dotnet. Use a comma-separated list to specify more than one language:

DD_APM_INSTRUMENTATION_LIBRARIES=java:1,js:5 DD_APM_INSTRUMENTATION_ENABLED=all DD_API_KEY=<YOUR KEY> DD_SITE="<YOUR SITE>" bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

Specifying Unified Service Tags on containers

If the environment variables DD_ENV, DD_SERVICE, or DD_VERSION are specified in a service container image, those values are used to tag telemetry from the container.

If they are not specified, DD_ENV uses the env value set in the /etc/datadog-agent/inject/docker_config.yaml config file, if any. DD_SERVICE is derived from the name of the Docker image. An image with the name my-service:1.0 is tagged with DD_SERVICE of my-service.

Launch your services

Start your Agent and launch your containerized services as usual.

Exercise your application to start generating telemetry data, which you can see as traces in APM.

Tracing Library Injection in containers is in beta.

When your Agent and services are running in separate Docker containers on the same host, Datadog injects the tracing library by intercepting container creation and configuring the Docker container.

Any newly started processes are intercepted and the specified instrumentation library is injected into the services.

Requirements:

Note: Injection on arm64 is not supported.

Install the preload library

Use the install_script_docker_injection shell script to automatically install Docker injection support. Docker must already be installed on the host machine.

DD_APM_INSTRUMENTATION_ENABLED=docker DD_APM_INSTRUMENTATION_LIBRARIES=java:1,python:2,js:5,dotnet:3,ruby:2 DD_NO_AGENT_INSTALL=true bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

This installs language libraries for all supported languages. To install specific languages, set the DD_APM_INSTRUMENTATION_LIBRARIES variable. The valid values are java, js, python, ruby, and dotnet:

DD_APM_INSTRUMENTATION_ENABLED=docker DD_APM_INSTRUMENTATION_LIBRARIES=java:1,js:5 DD_NO_AGENT_INSTALL=true bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

Configure the Agent

In the Docker compose file that launches your containers, use the following settings for the Agent, securely setting your own Datadog API key for ${DD_API_KEY}:

  dd-agent:
    container_name: dd-agent
    image: datadog/agent:7
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_APM_ENABLED=true
      - DD_APM_NON_LOCAL_TRAFFIC=true
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
      - DD_APM_RECEIVER_SOCKET=/var/run/datadog/apm.socket
      - DD_DOGSTATSD_SOCKET=/var/run/datadog/dsd.socket
    volumes:
      - /var/run/datadog:/var/run/datadog
      - /var/run/docker.sock:/var/run/docker.sock:ro

Specifying Unified Service Tags on containers

If the environment variables DD_ENV, DD_SERVICE, or DD_VERSION are specified in a service container image, those values are used to tag telemetry from the container.

If they are not specified, DD_ENV uses the env value set in the /etc/datadog-agent/inject/docker_config.yaml config file, if any. DD_SERVICE is derived from the name of the Docker image. An image with the name my-service:1.0 is tagged with DD_SERVICE of my-service.

Launch the Agent on Docker

The dd-agent container must be launched before any service containers. Run:

docker-compose up -d dd-agent

Launch your services

Launch your containerized services as usual.

Exercise your application to start generating telemetry data, which you can see as traces in APM.

Uninstall library injection

Remove instrumentation for specific services

To stop producing traces for a specific service, run the following commands and restart the service:

  1. Add the DD_INSTRUMENT_SERVICE_WITH_APM environment variable to the service startup command:

    DD_INSTRUMENT_SERVICE_WITH_APM=false <service_start_command>
    
  2. Restart the service.

  1. Add the DD_INSTRUMENT_SERVICE_WITH_APM environment variable to the service startup command:
    docker run -e DD_INSTRUMENT_SERVICE_WITH_APM=false
    
  2. Restart the service.

Remove APM for all services on the infrastructure

To stop producing traces, remove library injectors and restart the infrastructure:

  1. Run:
    dd-host-install --uninstall
    
  2. Restart your host.
  1. Uninstall local library injection:
    dd-container-install --uninstall
    
  2. Restart Docker:
    systemctl restart docker
    
    Or use the equivalent for your environment.

Configuring the library

The supported features and configuration options for the tracing library are the same for library injection as for other installation methods, and can be set with environment variables. Read the Datadog library configuration page for your language for more details.

For example, you can turn on Application Security Monitoring or Continuous Profiler, each of which may have billing impact:

  • For Kubernetes, set the DD_APPSEC_ENABLED or DD_PROFILING_ENABLED environment variables to true in the underlying application pod’s deployment file.

  • For hosts and containers, set the DD_APPSEC_ENABLED or DD_PROFILING_ENABLED container environment variables to true.