Single Step APM Instrumentation on Kubernetes
Overview
Single Step Instrumentation for Kubernetes is GA for Agent versions 7.64+, and in Preview for Agent versions <=7.63.
In a Kubernetes environment, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and [instrument][10] your applications in one step, with no additional configuration required.
Requirements
- Kubernetes v1.20+
Helm
for deploying the Datadog Operator.Kubectl
CLI for installing the Datadog Agent.
Enable APM on your applications
Single Step Instrumentation does not instrument applications in the namespace where the Datadog Agent is installed. Install the Agent in a separate namespace where you do not run your applications.
Follow these steps to enable Single Step Instrumentation across your entire cluster. This automatically sends traces from all applications written in supported languages.
Note: To instrument only specific namespaces or pods, see Advanced options.
In the Datadog app, go to the Install the Datadog Agent on Kubernetes page.
Follow the on-screen instructions to choose your installation method, select an API key, and set up the Operator or Helm repository.
In the Configure datadog-agent.yaml
section, go to Additional configuration > Application Observability, and turn on APM Instrumentation.
Deploy the Agent using the generated configuration file.
Restart your applications.
Advanced options
When you run the one-line installation command, there are a few options to customize your experience:
By default, Single Step Instrumentation instruments all services in all namespaces in your cluster. Alternatively, you can create targeting blocks with the targets
label to specify which workloads to instrument and what configurations to apply.
Each target block has the following keys:
Key | Description |
---|
name | The name of the target block. This has no effect on monitoring state and is used only as metadata. |
namespaceSelector | The namespace(s) to instrument. Specify using one or more of: - matchNames : A list of one or more namespace name(s). - matchLabels : A list of one or more label(s) defined in {key,value} pairs. - matchExpressions : A list of namespace selector requirements.
Namespaces must meet all criteria to match. For more details, see the Kubernetes selector documentation. |
podSelector | The pod(s) to instrument. Specify using one or more of: - matchLabels : A list of one or more label(s) defined in {key,value} pairs. - matchExpressions : A list of pod selector requirements.
Pods must meet all criteria to match. For more details, see the Kubernetes selector documentation. |
ddTraceVersions | The Datadog APM SDK version to use for each language. |
ddTraceConfigs | APM SDK configs that allow setting Unified Service Tags, enabling Datadog products beyond tracing, and customizing other APM settings. See full list of options. |
The file you need to configure depends on how you enabled Single Step Instrumentation:
- If you enabled SSI with Datadog Operator, edit
datadog-agent.yaml
. - If you enabled SSI with Helm, edit
datadog-values.yaml
.
Example configurations
Review the following examples demonstrating how to select specific services:
This configuration:
- enables APM for all namespaces except the
jenkins
namespace. - instructs Datadog to instrument the Java applications with the default Java APM SDK and Python applications with
v.3.1.0
of the Python APM SDK.
apm:
instrumentation:
enabled: true
disabledNamespaces:
- "jenkins"
targets:
- name: "all-remaining-services"
ddTraceVersions:
java: "default"
python: "3.1.0"
This configuration creates two targets blocks:
- The first block (named
login-service_namespace
):- enables APM for services in the namespace
login-service
. - instructs Datadog to instrument services in this namespace with the default version of the Java APM SDK.
- sets environment variable
DD_PROFILING_ENABLED
for this target group
- The second block (named
billing-service_apps
)- enables APM for services in the namespace(s) with label
app:billing-service
. - instructs Datadog to instrument this set of services with
v3.1.0
of the Python APM SDK.
apm:
instrumentation:
enabled: true
targets:
- name: "login-service_namespace"
namespaceSelector:
matchNames:
- "login-service"
ddTraceVersions:
java: "default"
ddTraceConfigs:
- name: "DD_PROFILING_ENABLED" ## profiling is enabled for all services in this namespace
value: "auto"
- name: "billing-service_apps"
namespaceSelector:
matchLabels:
app: "billing-service"
ddTraceVersions:
python: "3.1.0"
This configuration does the following:
- enables APM for pods with the following labels:
app:db-user
, which marks pods running the db-user
application.webserver:routing
, which marks pods running the request-router
application.
- instructs Datadog to use the default versions of the Datadog Tracer SDKs.
- sets Datadog environment variables to apply to each target group and configure the SDKs.
apm:
instrumentation:
enabled: true
targets:
- name: "db-user"
podSelector:
matchLabels:
app: "db-user"
ddTraceVersions:
java: "default"
ddTraceConfigs: ## trace configs set for services in matching pods
- name: "DD_DSM_ENABLED"
value: "true"
- name: "user-request-router"
podSelector:
matchLabels:
webserver: "user"
ddTraceVersions:
php: "default"
This configuration:
- enables APM for pods labeled
app:password-resolver
inside the login-service
namespace. - instructs Datadog to use the default version of the Datadog Java Tracer SDK.
- sets Datadog environment variables to apply to this target.
apm:
instrumentation:
enabled: true
targets:
- name: "login-service-namespace"
namespaceSelector:
matchNames:
- "login-service"
podSelector:
matchLabels:
app: "password-resolver"
ddTraceVersions:
java: "default"
ddTraceConfigs:
- name: "DD_PROFILING_ENABLED"
value: "auto"
This configuration enables APM for all pods except those that have either of the labels app=app1
or app=app2
.
apm:
instrumentation:
enabled: true
targets:
- name: "default-target"
matchExpressions:
- key: app
operator: NotIn
values:
- app1
- app2
Enable or disable instrumentation for namespaces
You can choose to enable or disable instrumentation for applications in specific namespaces. You can only set enabledNamespaces or disabledNamespaces, not both.
The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:
To enable instrumentation for specific namespaces, add enabledNamespaces
configuration to datadog-agent.yaml
:
features:
apm:
instrumentation:
enabled: true
enabledNamespaces: # Add namespaces to instrument
- default
- applications
To disable instrumentation for specific namespaces, add disabledNamespaces
configuration to datadog-agent.yaml
:
features:
apm:
instrumentation:
enabled: true
disabledNamespaces: # Add namespaces to not instrument
- default
- applications
To enable instrumentation for specific namespaces, add enabledNamespaces
configuration to datadog-values.yaml
:
datadog:
apm:
instrumentation:
enabled: true
enabledNamespaces: # Add namespaces to instrument
- namespace_1
- namespace_2
To disable instrumentation for specific namespaces, add disabledNamespaces
configuration to datadog-values.yaml
:
datadog:
apm:
instrumentation:
enabled: true
disabledNamespaces: # Add namespaces to not instrument
- namespace_1
- namespace_2
Specify tracing library versions
Starting with Datadog Cluster Agent v7.52.0+, you can automatically instrument a subset of your applications, based on the tracing libraries you specify.
Specify Datadog tracing libraries and their versions to automatically instrument applications written in those languages. You can configure this in two ways, which are applied in the following order of precedence:
- Specify at the service level, or
- Specify at the cluster level.
Default: If you don’t specify any library versions, applications written in supported languages are automatically instrumented using the latest tracing library versions.
Specify at the service level
To automatically instrument applications in specific pods, add the appropriate language annotation and library version for your application in your pod spec:
Language | Pod annotation |
---|
Java | admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>" |
Node.js | admission.datadoghq.com/js-lib.version: "<CONTAINER IMAGE TAG>" |
Python | admission.datadoghq.com/python-lib.version: "<CONTAINER IMAGE TAG>" |
.NET | admission.datadoghq.com/dotnet-lib.version: "<CONTAINER IMAGE TAG>" |
Ruby | admission.datadoghq.com/ruby-lib.version: "<CONTAINER IMAGE TAG>" |
PHP | admission.datadoghq.com/php-lib.version: "<CONTAINER IMAGE TAG>" |
Replace <CONTAINER IMAGE TAG>
with the desired library version. Available versions are listed in the Datadog container registries and tracer source repositories for each language:
Exercise caution when using the latest
tag, as major library releases may introduce breaking changes.
For example, to automatically instrument Java applications:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
# ...
spec:
template:
metadata:
annotations:
admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
spec:
containers:
- # ...
Specify at the cluster level
If you don’t enable automatic instrumentation for specific pods using annotations, you can specify which languages to instrument across the entire cluster using the Single Step Instrumentation configuration. When apm.instrumentation.libVersions
is set, only applications written in the specified languages will be instrumented, using the specified library versions.
The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:
For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-agent.yaml
file:
features:
apm:
instrumentation:
enabled: true
libVersions: # Add any libraries and versions you want to set
dotnet: "3.2.0"
python: "1.20.6"
js: "4.17.0"
For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-values.yaml
file:
datadog:
apm:
instrumentation:
enabled: true
libVersions: # Add any libraries and versions you want to set
dotnet: "3.2.0"
python: "1.20.6"
js: "4.17.0"
Container registries
Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and Amazon ECR:
The DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY
environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is gcr.io/datadoghq
.
You can pull the tracing library from a different registry by changing it to docker.io/datadog
, public.ecr.aws/datadog
, or another URL if you are hosting the images in a local container registry.
For instructions on changing your container registry, see Changing Your Container Registry.
Remove Single Step APM instrumentation from your Agent
If you don’t want to collect trace data for a particular service, host, VM, or container, complete the following steps:
Remove instrumentation for specific services
To remove APM instrumentation and stop sending traces from a specific service, you can do one of the following:
Use workload selection (recommended)
With workload selection (available for Agent v7.64+), you can enable and disable tracing for specific applications. See configuration details here.
Use the Datadog Admission Controller
As an alternative, or for a version of the agent that does not support workload selection, you can also disable pod mutation by adding a label to your pod.
In addition to disabling SSI, the following steps disable other mutating webhooks. Use with caution.
- Set the
admission.datadoghq.com/enabled:
label to "false"
for the pod spec:spec:
template:
metadata:
labels:
admission.datadoghq.com/enabled: "false"
- Apply the configuration:
kubectl apply -f /path/to/your/deployment.yaml
- Restart the services you want to remove instrumentation for.
Remove APM for all services on the infrastructure
To stop producing traces, uninstall APM and restart the infrastructure:
The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:
Set instrumentation.enabled=false
in datadog-agent.yaml
:
features:
apm:
instrumentation:
enabled: false
Deploy the Datadog Agent with the updated configuration file:
kubectl apply -f /path/to/your/datadog-agent.yaml
Set instrumentation.enabled=false
in datadog-values.yaml
:
datadog:
apm:
instrumentation:
enabled: false
Run the following command:
helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog
Further reading
Additional helpful documentation, links, and articles: