DD_APM_INSTRUMENTATION_LIBRARIES
- customizing APM libraries
By default, Java, Python, Ruby, Node.js, PHP and .NET Core Datadog APM libraries are installed when DD_APM_INSTRUMENTATION_ENABLED
is set. DD_APM_INSTRUMENTATION_LIBRARIES
is used to override which libraries are installed. The value is a comma-separated string of colon-separated library name and version pairs.
Example values for DD_APM_INSTRUMENTATION_LIBRARIES
:
DD_APM_INSTRUMENTATION_LIBRARIES="java:1"
- install only the Java Datadog APM library pinned to the major version 1 release line.DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:2"
- install only the Java and Python Datadog APM libraries pinned to the major versions 1 and 2 respectively.DD_APM_INSTRUMENTATION_LIBRARIES="java:1.38.0,python:2.10.5"
- install only the Java and Python Datadog APM libraries pinned to the specific versions 1.38.0 and 2.10.5 respectively.
Available versions are listed in source repositories for each language:
DD_APM_INSTRUMENTATION_LIBRARIES
- customizing APM libraries
By default, Java, Python, Ruby, Node.js and .NET Core Datadog APM libraries are installed when DD_APM_INSTRUMENTATION_ENABLED
is set. DD_APM_INSTRUMENTATION_LIBRARIES
is used to override which libraries are installed. The value is a comma-separated string of colon-separated library name and version pairs.
Example values for DD_APM_INSTRUMENTATION_LIBRARIES
:
DD_APM_INSTRUMENTATION_LIBRARIES="java:1"
- install only the Java Datadog APM library pinned to the major version 1 release line.DD_APM_INSTRUMENTATION_LIBRARIES="java:1,python:2"
- install only the Java and Python Datadog APM libraries pinned to the major versions 1 and 2 respectively.DD_APM_INSTRUMENTATION_LIBRARIES="java:1.38.0,python:2.10.5"
- install only the Java and Python Datadog APM libraries pinned to the specific versions 1.38.0 and 2.10.5 respectively.
Available versions are listed in source repositories for each language:
Configuring instrumentation for namespaces and pods
By default, Single Step Instrumentation instruments all services in all namespaces in your cluster. Alternatively, you can create targeting blocks with the targets
label to specify which workloads to instrument and what configurations to apply.
Each target block has the following keys:
Key | Description |
---|
name | The name of the target block. This has no effect on monitoring state and is used only as metadata. |
namespaceSelector | The namespace(s) to instrument. Specify using one or more of: - matchNames : A list of one or more namespace name(s). - matchLabels : A list of one or more label(s) defined in {key,value} pairs. - matchExpressions : A list of namespace selector requirements.
Namespaces must meet all criteria to match. For more details, see the Kubernetes selector documentation. |
podSelector | The pod(s) to instrument. Specify using one or more of: - matchLabels : A list of one or more label(s) defined in {key,value} pairs. - matchExpressions : A list of pod selector requirements.
Pods must meet all criteria to match. For more details, see the Kubernetes selector documentation. |
ddTraceVersions | The Datadog APM SDK version to use for each language. |
ddTraceConfigs | APM SDK configs that allow setting Unified Service Tags, enabling Datadog products beyond tracing, and customizing other APM settings. See full list of options. |
The file you need to configure depends on how you enabled Single Step Instrumentation:
- If you enabled SSI with Datadog Operator, edit
datadog-agent.yaml
. - If you enabled SSI with Helm, edit
datadog-values.yaml
.
Example configurations
Review the following examples demonstrating how to select specific services:
This configuration:
- enables APM for all namespaces except the
jenkins
namespace. - instructs Datadog to instrument the Java applications with the default Java APM SDK and Python applications with
v.3.1.0
of the Python APM SDK.
apm:
instrumentation:
enabled: true
disabledNamespaces:
- "jenkins"
targets:
- name: "all-remaining-services"
ddTraceVersions:
java: "default"
python: "3.1.0"
This configuration creates two targets blocks:
- The first block (named
login-service_namespace
):- enables APM for services in the namespace
login-service
. - instructs Datadog to instrument services in this namespace with the default version of the Java APM SDK.
- sets environment variables –
DD_SERVICE
, DD_ENV
, and DD_PROFILING_ENABLED
– for this target group.
- The second block (named
billing-service_apps
)- enables APM for services in the namespace(s) with label
app:billing-service
. - instructs Datadog to instrument this set of services with
v3.1.0
of the Python APM SDK. - sets environment variables –
DD_SERVICE
and DD_ENV
– for this target group.
apm:
instrumentation:
enabled: true
targets:
- name: "login-service_namespace"
namespaceSelector:
matchNames:
- "login-service"
ddTraceVersions:
java: "default"
ddTraceConfigs:
- name: "DD_SERVICE"
value: "login-service"
- name: "DD_ENV"
value: "prod"
- name: "DD_PROFILING_ENABLED" ## profiling is enabled for all services in this namespace
value: "auto"
- name: "billing-service_apps"
namespaceSelector:
matchLabels:
app: "billing-service"
ddTraceVersions:
python: "3.1.0"
ddTraceConfigs:
- name: "DD_SERVICE"
value: "billing-service"
- name: "DD_ENV"
value: "prod
This configuration does the following:
- enables APM for pods with the following labels:
app:db-user
, which marks pods running the db-user
application.webserver:routing
, which marks pods running the request-router
application.
- instructs Datadog to use the default versions of the Datadog Tracer SDKs.
- sets several Datadog environment variables to apply to each target group.
apm:
instrumentation:
enabled: true
targets:
- name: "db-user"
podSelector:
matchLabels:
app: "db-user"
ddTraceVersions:
java: "default"
ddTraceConfigs: ## trace configs set for services in matching pods
- name: "DD_SERVICE"
value: "db-user"
- name: "DD_ENV"
value: "prod"
- name: "DD_DSM_ENABLED"
value: "true"
- name: "user-request-router"
podSelector:
matchLabels:
webserver: "user"
ddTraceVersions:
php: "default"
ddTraceConfigs:
- name: "DD_SERVICE"
value: "user-request-router"
- name: "DD_ENV"
value: "prod
This configuration:
- enables APM for pods labeled
app:password-resolver
inside the login-service
namespace. - instructs Datadog to use the default version of the Datadog Java Tracer SDK.
- sets several Datadog environment variables to apply to this target.
apm:
instrumentation:
enabled: true
targets:
- name: "login-service-namespace"
namespaceSelector:
matchNames:
- "login-service"
podSelector:
matchLabels:
app: "password-resolver"
ddTraceVersions:
java: "default"
ddTraceConfigs:
- name: "DD_SERVICE"
value: "password-resolver"
- name: "DD_ENV"
value: "prod"
- name: "DD_PROFILING_ENABLED"
value: "auto"
This configuration enables APM for all pods except those that have either of the labels app=app1
or app=app2
.
apm:
instrumentation:
enabled: true
targets:
- name: "default-target"
matchExpressions:
- key: app
operator: NotIn
values:
- app1
- app2
Enabling or disabling instrumentation for namespaces
You can choose to enable or disable instrumentation for applications in specific namespaces. You can only set enabledNamespaces or disabledNamespaces, not both.
The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:
To enable instrumentation for specific namespaces, add enabledNamespaces
configuration to datadog-agent.yaml
:
features:
apm:
instrumentation:
enabled: true
enabledNamespaces: # Add namespaces to instrument
- default
- applications
To disable instrumentation for specific namespaces, add disabledNamespaces
configuration to datadog-agent.yaml
:
features:
apm:
instrumentation:
enabled: true
disabledNamespaces: # Add namespaces to not instrument
- default
- applications
To enable instrumentation for specific namespaces, add enabledNamespaces
configuration to datadog-values.yaml
:
datadog:
apm:
instrumentation:
enabled: true
enabledNamespaces: # Add namespaces to instrument
- namespace_1
- namespace_2
To disable instrumentation for specific namespaces, add disabledNamespaces
configuration to datadog-values.yaml
:
datadog:
apm:
instrumentation:
enabled: true
disabledNamespaces: # Add namespaces to not instrument
- namespace_1
- namespace_2
Specifying tracing library versions
Starting with Datadog Cluster Agent v7.52.0+, you can automatically instrument a subset of your applications, based on the tracing libraries you specify.
Specify Datadog tracing libraries and their versions to automatically instrument applications written in those languages. You can configure this in two ways, which are applied in the following order of precedence:
- Specify at the service level, or
- Specify at the cluster level.
Default: If you don’t specify any library versions and apm.instrumentation.enabled=true
, applications written in supported languages are automatically instrumented using the latest tracing library versions.
Specifying at the service level
To automatically instrument applications in specific pods, add the appropriate language annotation and library version for your application in your pod spec:
Language | Pod annotation |
---|
Java | admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>" |
Node.js | admission.datadoghq.com/js-lib.version: "<CONTAINER IMAGE TAG>" |
Python | admission.datadoghq.com/python-lib.version: "<CONTAINER IMAGE TAG>" |
.NET | admission.datadoghq.com/dotnet-lib.version: "<CONTAINER IMAGE TAG>" |
Ruby | admission.datadoghq.com/ruby-lib.version: "<CONTAINER IMAGE TAG>" |
PHP | admission.datadoghq.com/php-lib.version: "<CONTAINER IMAGE TAG>" |
Replace <CONTAINER IMAGE TAG>
with the desired library version. Available versions are listed in the Datadog container registries and tracer source repositories for each language:
Exercise caution when using the latest
tag, as major library releases may introduce breaking changes.
For example, to automatically instrument Java applications:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
# ...
spec:
template:
metadata:
annotations:
admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
spec:
containers:
- # ...
Specifying at the cluster level
If you don’t enable automatic instrumentation for specific pods using annotations, you can specify which languages to instrument across the entire cluster using the Single Step Instrumentation configuration. When apm.instrumentation.libVersions
is set, only applications written in the specified languages will be instrumented, using the specified library versions.
The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:
For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-agent.yaml
file:
features:
apm:
instrumentation:
enabled: true
libVersions: # Add any libraries and versions you want to set
dotnet: "3.2.0"
python: "1.20.6"
js: "4.17.0"
For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-values.yaml
file:
datadog:
apm:
instrumentation:
enabled: true
libVersions: # Add any libraries and versions you want to set
dotnet: "3.2.0"
python: "1.20.6"
js: "4.17.0"
Container registries
Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and Amazon ECR:
The DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY
environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is gcr.io/datadoghq
.
You can pull the tracing library from a different registry by changing it to docker.io/datadog
, public.ecr.aws/datadog
, or another URL if you are hosting the images in a local container registry.
For instructions on changing your container registry, see Changing Your Container Registry.