Injecting Libraries Using Admission Controller
Overview
In a Kubernetes environment, there are two ways to instrument your application:
With the Admission Controller approach, the Agent uses the Kubernetes Admission Controller to intercept requests to the Kubernetes API and mutate new pods to inject the specified instrumentation library.
Library injection is applied on new pods only and does not have any impact on running pods.
To learn more about Kubernetes Admission Controller, read Kubernetes Admission Controllers Reference.
Requirements
- Kubernetes v1.14+
- Datadog Cluster Agent v7.40+ with Datadog Admission Controller enabled. Note: In Helm chart v2.35.0 and later, Datadog Admission Controller is activated by default in the Cluster Agent.
- Applications in Java, JavaScript, or Python deployed on Linux with a supported architecture. Check the corresponding container registry for the complete list of supported architectures by language.
Container registries
Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and AWS ECR:
The DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY
environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is grc.io/datadoghq
.
You can pull the tracing library from a different registry by changing it to docker.io/datadog
, public.ecr.aws/datadog
, or another URL if you are hosting the images in a local container registry.
For your Kubernetes applications whose traces you want to send to Datadog, configure the Datadog Admission Controller to inject Java, JavaScript, or Python instrumentation libraries automatically. From a high level, this involves the following steps, described in detail below:
- Enable Datadog Admission Controller to mutate your pods.
- Annotate your pods to select which instrumentation library to inject.
- Tag your pods with Unified Service Tags to tie Datadog telemetry together and navigate seamlessly across traces, metrics, and logs with consistent tags.
- Apply your new configuration.
You do not need to generate a new application image to inject the library. The library injection is taking care of adding the instrumentation library, so no change is required in your application image.
Step 1 - Enable Datadog Admission Controller to mutate your pods
By default, Datadog Admission controller mutates only pods labeled with a specific label. To enable mutation on your pods, add the label admission.datadoghq.com/enabled: "true"
to your pod spec.
Note: You can configure Datadog Admission Controller to enable injection config without having this pod label by configuring the Cluster Agent with clusterAgent.admissionController.mutateUnlabelled
(or DD_ADMISSION_CONTROLLER_MUTATE_UNLABELLED
) to true
.
For more details on how to configure, read Datadog Admission Controller page.
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
...
...
template:
metadata:
labels:
admission.datadoghq.com/enabled: "true" # Enable Admission Controller to mutate new pods part of this deployment
containers:
- ...
Step 2 - Annotate your pods for library injection
To select your pods for library injection, annotate them with the following, corresponding to your application language, in your pod spec:
Language | Pod annotation |
---|
Java | admission.datadoghq.com/java-lib.version: "<lib-version>" |
JavaScript | admission.datadoghq.com/js-lib.version: "<lib-version>" |
Python | admission.datadoghq.com/python-lib.version: "<lib-version>" |
The available library versions are listed in each container registry.
Note: If you already have an application instrumented using version X of the library, and then use library injection to instrument using version Y of the same tracer library, the tracer does not break. Rather, the library version loaded first is used. As Library Injection happens at the admission controller level prior to runtime, it will take precedent over any manually configured libraries.
Note: Using the latest
tag is supported, but use it with caution because major library releases can introduce breaking changes.
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
...
...
template:
metadata:
labels:
admission.datadoghq.com/enabled: "true" # Enable Admission Controller to mutate new pods in this deployment
annotations:
admission.datadoghq.com/java-lib.version: "v0.114.0" # Enable Java instrumentation (version 0.114.0) injection
containers:
- ...
With Unified Service Tags, you can tie Datadog telemetry together and navigate seamlessly across traces, metrics, and logs with consistent tags. Set the Unified Service Tagging on both the deployment object and the pod template specs.
Set Unified Service tags by using the following labels:
...
metadata:
labels:
tags.datadoghq.com/env: "<ENV>"
tags.datadoghq.com/service: "<SERVICE>"
tags.datadoghq.com/version: "<VERSION>"
...
Note: It is not necessary to set the environment variables for universal service tagging (DD_ENV
, DD_SERVICE
, DD_VERSION
) in the pod template spec, because the Admission Controller propagates the tag values as environment variables when injecting the library.
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
tags.datadoghq.com/env: "prod" # Unified service tag - Deployment Env tag
tags.datadoghq.com/service: "my-service" # Unified service tag - Deployment Service tag
tags.datadoghq.com/version: "1.1" # Unified service tag - Deployment Version tag
...
template:
metadata:
labels:
tags.datadoghq.com/env: "prod" # Unified service tag - Pod Env tag
tags.datadoghq.com/service: "my-service" # Unified service tag - Pod Service tag
tags.datadoghq.com/version: "1.1" # Unified service tag - Pod Version tag
admission.datadoghq.com/enabled: "true" # Enable Admission Controller to mutate new pods part of this deployment
annotations:
admission.datadoghq.com/java-lib.version: "v0.114.0" # Enable Java instrumentation (version 0.114.0) injection
containers:
- ...
Step 4 - Apply the configuration
Your pods are ready to be instrumented when their new configuration is applied.
The library is injected on new pods only and does not have any impact on running pods.
Check that the library injection was successful
Library injection leverages the injection of a dedicated init
container in pods.
If the injection was successful you can see an init
container called datadog-lib-init
in your pod:
Or run kubectl describe pod <my-pod>
to see the datadog-lib-init
init container listed.
The instrumentation also starts sending telemetry to Datadog (for example, traces to APM).
Configuring the library
The supported features and configuration options for the tracing library are the same for library injection as for other installation methods, and can be set with environment variables. Read the Datadog Library configuration page for your language for more details.