Single Step APM Instrumentation (Beta)

Overview

Single Step Instrumentation for APM installs the Datadog Agent and instruments your applications in one step, with no additional configuration steps required.

Requirements

  • Languages and architectures: Single step APM instrumentation only supports tracing Java, Python, Ruby, Node.js, and .NET Core services on x86_64 and arm64 architectures.

  • Operating systems: Linux VMs (Debian, Ubuntu, Amazon Linux, CentOS/Red Hat, Fedora), Docker, Kubernetes clusters with Linux containers.

Enabling APM on your applications

If you install or update a Datadog Agent with the Enable APM Instrumentation (beta) option selected, the Agent is installed and configured to enable APM. This automatically instruments your application, without any additional installation or configuration steps.

The following examples show how it works for each deployment type.

If you've previously used Single Step Instrumentation with Linux hosts, update to the latest version.

For an Ubuntu host:

  1. Run the one-line installation command:

    DD_API_KEY=<YOUR_DD_API_KEY> DD_SITE="<YOUR_DD_SITE>" DD_APM_INSTRUMENTATION_ENABLED=host DD_ENV=<AGENT_ENV> bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"
    

    Replace <YOUR_DD_API_KEY> with your Datadog API key, <YOUR_DD_SITE> with your Datadog site, and <AGENT_ENV> with the environment your Agent is installed on (for example, env:staging).

    See Advanced options for more options.
  2. Start a new shell session.

  3. Restart the services on the host or VM.

For a Docker Linux container:

  1. Run the one-line installation command:
    bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_docker_injection.sh)"
    
  2. Configure the Agent in Docker:
    docker run -d --name dd-agent \
      -e DD_API_KEY=${YOUR_DD_API_KEY} \
      -e DD_APM_ENABLED=true \
      -e DD_ENV=<AGENT_ENV>
      -e DD_APM_NON_LOCAL_TRAFFIC=true \
      -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
      -e DD_APM_RECEIVER_SOCKET=/opt/datadog/apm/inject/run/apm.socket \
      -e DD_DOGSTATSD_SOCKET=/opt/datadog/apm/inject/run/dsd.socket \
      -v /opt/datadog/apm:/opt/datadog/apm \
      -v /var/run/docker.sock:/var/run/docker.sock:ro \
      gcr.io/datadoghq/agent:7
    
    Replace <YOUR_DD_API_KEY> with your Datadog API key and <AGENT_ENV> with the environment your Agent is installed on (for example, env:staging).
    See Advanced options for more options.
  3. Restart the Docker containers.
  4. Explore the performance observability of your services in Datadog.

You can enable APM by installing the Agent with either:

  • Datadog Operator
  • Datadog Helm chart
Single Step Instrumentation doesn't instrument applications in the namespace where you install the Datadog Agent. It's recommended to install the Agent in a separate namespace in your cluster where you don't run your applications.

Requirements

  • Kubernetes v1.20+
  • Helm for deploying the Datadog Operator.
  • Kubectl CLI for installing the Datadog Agent.

Follow these steps to enable Single Step Instrumentation across your entire cluster with the Datadog Operator. This automatically sends traces for all applications in the cluster that are written in supported languages.

To enable Single Step Instrumentation with the Datadog Operator:

  1. Install the Datadog Operator v1.5.0+ with Helm:

    helm repo add datadog https://helm.datadoghq.com
    helm install my-datadog-operator datadog/datadog-operator
    
  2. Create a Kubernetes secret to store your Datadog API key:

    kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY
    
  3. Create datadog-agent.yaml with the spec of your Datadog Agent deployment configuration. The simplest configuration is as follows:

    apiVersion: datadoghq.com/v2alpha1
    kind: DatadogAgent
    metadata:
      name: datadog
    spec:
      global:
        site: <DATADOG_SITE>
        tags:
          - env:<AGENT_ENV>
        credentials:
          apiSecret:
            secretName: datadog-secret
            keyName: api-key
      features:
        apm:
          instrumentation:
            enabled: true  
    

    Replace <DATADOG_SITE> with your Datadog site and <AGENT_ENV> with the environment your Agent is installed on (for example, env:staging).

    See Advanced options for more options.
  4. Run the following command:

    kubectl apply -f /path/to/your/datadog-agent.yaml
    
  5. After waiting a few minutes for the Datadog Cluster Agent changes to apply, restart your applications.

Follow these steps to enable Single Step Instrumentation across your entire cluster with Helm. This automatically sends traces for all applications in the cluster that are written in supported languages.

To enable Single Step Instrumentation with Helm:

  1. Add the Helm Datadog repo:

     helm repo add datadog https://helm.datadoghq.com
     helm repo update
    
  2. Create a Kubernetes secret to store your Datadog API key:

    kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY
    
  3. Create datadog-values.yaml and add the following configuration:

    datadog:
     apiKeyExistingSecret: datadog-secret
     site: <DATADOG_SITE>
     tags:
          - env:<AGENT_ENV>
     apm:
       instrumentation:
          enabled: true
    

    Replace <DATADOG_SITE> with your Datadog site and <AGENT_ENV> with the environment your Agent is installed on (for example, env:staging).

    See Advanced options for more options.
  4. Run the following command:

    helm install datadog-agent -f datadog-values.yaml datadog/datadog
    
  5. After waiting a few minutes for the Datadog Cluster Agent changes to apply, restart your applications.

After you complete these steps, you may want to enable runtime metrics or view observability data from your application in the Service Catalog.

Advanced options

When you run the one-line installation command, there are a few options to customize your experience:

Specifying tracing library versions

By default, enabling APM on your server installs support for Java, Python, Ruby, Node.js, and .NET Core services. If you only have services implemented in some of these languages, set DD_APM_INSTRUMENTATION_LIBRARIES in your one-line installation command.

For example, to install support for only v1.25.0 of the Java tracing library and the latest Python tracing library, add the following to the installation command:

DD_APM_INSTRUMENTATION_LIBRARIES="java:1.25.0,python" DD_API_KEY=<YOUR_DD_API_KEY> DD_SITE="<YOUR_DD_SITE>" DD_APM_INSTRUMENTATION_ENABLED=host DD_ENV=staging bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

You can optionally provide a version number for the tracing library by placing a colon after the language name and specifying the tracing library version. If you don’t specify a version, it defaults to the latest version. Language names are comma-separated.

Available versions are listed in tracer source repositories for each language:

Specifying tracing library versions

By default, enabling APM on your server installs support for Java, Python, Ruby, Node.js, and .NET services. If you only have services implemented in some of these languages, set DD_APM_INSTRUMENTATION_LIBRARIES when running the installation script.

For example, to install support for only v1.25.0 of the Java tracing library and the latest Python tracing library, add the following to the installation command:

DD_APM_INSTRUMENTATION_LIBRARIES="java:1.25.0,python" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_docker_injection.sh)"

You can optionally provide a version number for the tracing library by placing a colon after the language name and specifying the tracing library version. If you don’t specify a version, it defaults to the latest version. Language names are comma-separated.

Available versions are listed in tracer source repositories for each language:

Enabling or disabling instrumentation for namespaces

You can choose to enable or disable instrumentation for applications in specific namespaces. You can only set enabledNamespaces or disabledNamespaces, not both.

The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:

To enable instrumentation for specific namespaces, add enabledNamespaces configuration to datadog-agent.yaml:

   features:
     apm:
       instrumentation:
         enabled: true 
         enabledNamespaces: # Add namespaces to instrument
           - default
           - applications

To disable instrumentation for specific namespaces, add disabledNamespaces configuration to datadog-agent.yaml:

   features:
     apm:
       instrumentation:
         enabled: true 
         disabledNamespaces: # Add namespaces to not instrument
           - default
           - applications

To enable instrumentation for specific namespaces, add enabledNamespaces configuration to datadog-values.yaml:

   datadog:
      apm:
        instrumentation:
          enabled: true
          enabledNamespaces: # Add namespaces to instrument
             - namespace_1
             - namespace_2

To disable instrumentation for specific namespaces, add disabledNamespaces configuration to datadog-values.yaml:

   datadog:
      apm:
        instrumentation:
          enabled: true
          disabledNamespaces: # Add namespaces to not instrument
            - namespace_1
            - namespace_2

Specifying tracing library versions

Starting with Datadog Cluster Agent v7.52.0+, you can automatically instrument a subset of your applications, based on the tracing libraries you specify.

Specify Datadog tracing libraries and their versions to automatically instrument applications written in those languages. You can configure this in two ways, which are applied in the following order of precedence:

  1. Specify at the service level, or
  2. Specify at the cluster level.

Default: If you don’t specify any library versions and apm.instrumentation.enabled=true, applications written in supported languages are automatically instrumented using the latest tracing library versions.

Specifying at the service level

To automatically instrument applications in specific pods, add the appropriate language annotation and library version for your application in your pod spec:

LanguagePod annotation
Javaadmission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
Node.jsadmission.datadoghq.com/js-lib.version: "<CONTAINER IMAGE TAG>"
Pythonadmission.datadoghq.com/python-lib.version: "<CONTAINER IMAGE TAG>"
.NETadmission.datadoghq.com/dotnet-lib.version: "<CONTAINER IMAGE TAG>"
Rubyadmission.datadoghq.com/ruby-lib.version: "<CONTAINER IMAGE TAG>"

Replace <CONTAINER IMAGE TAG> with the desired library version. Available versions are listed in the Datadog container registries and tracer source repositories for each language:

  • Java
  • Node.js
  • Python
  • .NET (For .NET applications using a musl-based Linux distribution like Alpine, specify a tag with the -musl suffix, such as v2.29.0-musl.)
  • Ruby
Exercise caution when using the latest tag, as major library releases may introduce breaking changes.

For example, to automatically instrument Java applications:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    # ...
spec:
  template:
    metadata:
      annotations:
        admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
    spec:
      containers:
        - # ...

Specifying at the cluster level

If you don’t enable automatic instrumentation for specific pods using annotations, you can specify which languages to instrument across the entire cluster using the Single Step Instrumentation configuration. When apm.instrumentation.libVersions is set, only applications written in the specified languages will be instrumented, using the specified library versions.

The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:

For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-agent.yaml file:

   features:
     apm:
       instrumentation:
         enabled: true
         libVersions: # Add any libraries and versions you want to set
            dotnet: v2.46.0
            python: v1.20.6
            js: v4.17.0

For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-values.yaml file:

   datadog:
     apm:
       instrumentation:
         enabled: true
         libVersions: # Add any libraries and versions you want to set
            dotnet: v2.46.0
            python: v1.20.6
            js: v4.17.0

Container registries

Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and Amazon ECR:

The DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is gcr.io/datadoghq.

You can pull the tracing library from a different registry by changing it to docker.io/datadog, public.ecr.aws/datadog, or another URL if you are hosting the images in a local container registry.

For instructions on changing your container registry, see Changing Your Container Registry.

Removing Single Step APM instrumentation from your Agent

If you don’t want to collect trace data for a particular service, host, VM, or container, complete the following steps:

Removing instrumentation for specific services

To remove APM instrumentation and stop sending traces from a specific service, follow these steps:

  1. Add the DD_INSTRUMENT_SERVICE_WITH_APM environment variable to the service startup command:

    DD_INSTRUMENT_SERVICE_WITH_APM=false <service_start_command>
    
  2. Restart the service.

  1. Add the DD_INSTRUMENT_SERVICE_WITH_APM environment variable to the service startup command:
    docker run -e DD_INSTRUMENT_SERVICE_WITH_APM=false <service_start_command>
    
  2. Restart the service.
  1. Set the admission.datadoghq.com/enabled: label to "false" for the pod spec:
    spec:
      template:
        metadata:
          labels:
            admission.datadoghq.com/enabled: "false"
    
  2. Apply the configuration:
    kubectl apply -f /path/to/your/deployment.yaml
    
  3. Restart the services you want to remove instrumentation for.

Removing APM for all services on the infrastructure

To stop producing traces, uninstall APM and restart the infrastructure:

  1. Run:
    dd-host-install --uninstall
    
  2. Restart your host.
  1. Run:
    dd-container-install --uninstall
    
  2. Restart Docker:
    systemctl restart docker
    
    Or use the equivalent for your environment.

The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:

  1. Set instrumentation.enabled=false in datadog-agent.yaml:

    features:
      apm:
        instrumentation:
          enabled: false
    
  2. Deploy the Datadog Agent with the updated configuration file:

    kubectl apply -f /path/to/your/datadog-agent.yaml
    
  1. Set instrumentation.enabled=false in datadog-values.yaml:

    datadog:
      apm:
        instrumentation:
          enabled: false
    
  2. Run the following command:

    helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog
    

Further reading

Additional helpful documentation, links, and articles: