Tracing Java Applications

Compatibility Requirements

The Java Tracing Library supports all JVMs on all platforms version 7 and higher. To utilize tracing with the Continuous Profiler, OpenJDK 11+, Oracle Java 11+, OpenJDK 8 for most vendors (version 8u262+) and Zulu Java 8+ (minor version 1.8.0_212+) are supported. Starting in version 8u272+, all vendors will be supported for the Profiler.

All JVM-based languages, such as Scala (versions 2.10.x - 2.13.x), Groovy, Kotlin, and Clojure are supported. For a full list of supported libraries, visit the Compatibility Requirements page.

Installation and Getting Started

Follow the in-app Documentation (Recommended)

Follow the Quickstart instructions within the Datadog app for the best experience, including:

• Step-by-step instructions scoped to your deployment configuration (hosts, Docker, Kubernetes, or Amazon ECS).
• Dynamically set service, env and version tags.
• Enable the Continuous Profiler, ingesting 100% of traces, and Trace ID injection into logs during setup.

Java Installation Steps

Otherwise, to begin tracing applications written in any language:

1. Download dd-java-agent.jar that contains the Agent class files:

   wget -O dd-java-agent.jar https://dtdg.co/latest-java-tracer
   

2. Add the following JVM argument when starting your application in your IDE, Maven or Gradle application script, or java -jar command:

    -javaagent:/path/to/the/dd-java-agent.jar
   

3. Add configuration options for tracing and ensure you are setting environment variables or passing system properties as JVM arguments, particularly for service, environment, logs injection, profiling, and optionally runtime metrics-all the metrics you intend to use. See the examples below. Note that using the in-app quickstart instructions generates these for you.

Configure the Datadog Agent for APM

Install and configure the Datadog Agent to receive traces from your now instrumented application. By default the Datadog Agent is enabled in your datadog.yaml file under apm_enabled: true and listens for trace traffic at localhost:8126. For containerized environments, follow the links below to enable trace collection within the Datadog Agent.

• Dropdown

1. Set apm_non_local_traffic: true in your main datadog.yaml configuration file

2. See the specific setup instructions to ensure that the Agent is configured to receive traces in a containerized environment:

3. After having instrumented your application, the tracing client sends traces to localhost:8126 by default. If this is not the correct host and port change it by setting the below env variables:

DD_AGENT_HOST and DD_TRACE_AGENT_PORT.

java -javaagent:<DD-JAVA-AGENT-PATH>.jar -jar <YOUR_APPLICATION_PATH>.jar

You can also use system properties:

java -javaagent:<DD-JAVA-AGENT-PATH>.jar \
     -Ddd.agent.host=$DD_AGENT_HOST \
     -Ddd.agent.port=$DD_TRACE_AGENT_PORT \
     -jar <YOUR_APPLICATION_PATH>.jar

To set up Datadog APM in AWS Lambda, see the Tracing Serverless Functions documentation.

Tracing is available for a number of other environments, such as Heroku, Cloud Foundry, AWS Elastic Beanstalk, and Azure App Services Extension.

For other environments, please refer to the Integrations documentation for that environment and contact support if you are encountering any setup issues.

Add the Java Tracer to the JVM

Use the documentation for your IDE to figure out the right way to pass in -javaagent and other JVM arguments. Here are instructions for some commonly used frameworks:

JAVA_OPTS=-javaagent:/path/to/dd-java-agent.jar

CATALINA_OPTS="$CATALINA_OPTS -javaagent:/path/to/dd-java-agent.jar"

set CATALINA_OPTS_OPTS=%CATALINA_OPTS_OPTS% -javaagent:"c:\path\to\dd-java-agent.jar"

JAVA_OPTS="$JAVA_OPTS -javaagent:/path/to/dd-java-agent.jar"

set "JAVA_OPTS=%JAVA_OPTS% -javaagent:X:/path/to/dd-java-agent.jar"

JAVA_OPTIONS="${JAVA_OPTIONS} -javaagent:/path/to/dd-java-agent.jar"

-javaagent:/path/to/dd-java-agent.jar

-javaagent:/path/to/dd-java-agent.jar

Note

• If you’re adding the -javaagent argument to your java -jar command, it needs to be added before the -jar argument, that is as a JVM option, not as an application argument. For example:

  java -javaagent:/path/to/dd-java-agent.jar -jar my_app.jar
  

  For more information, see the Oracle documentation.

• Never add dd-java-agent to your classpath. It can cause unexpected behavior.

Automatic Instrumentation

Automatic instrumentation for Java uses the java-agent instrumentation capabilities provided by the JVM. When a java-agent is registered, it has the ability to modify class files at load time.

Instrumentation may come from auto-instrumentation, the OpenTracing api, or a mixture of both. Instrumentation generally captures the following info:

• Timing duration is captured using the JVM’s nanotime clock unless a timestamp is provided from the OpenTracing API
• Key/value tag pairs
• Errors and stacktraces which are unhandled by the application
• A total count of traces (requests) flowing through the system

Configuration

All configuration options below have system property and environment variable equivalents. If the same key type is set for both, the system property configuration takes priority. System properties can be set as JVM flags.

Note:

• If the same key type is set for both, the system property configuration takes priority.

• System properties can be used as JVM parameters.

• By default, JMX metrics from your application are sent to the Datadog Agent thanks to DogStatsD over port 8125. Make sure that DogStatsD is enabled for the Agent.

  • If you are running the Agent as a container, ensure that DD_DOGSTATSD_NON_LOCAL_TRAFFIC is set to true, and that port 8125 is open on the Agent container.
  • In Kubernetes, bind the DogStatsD port to a host port; in ECS, set the appropriate flags in your task definition.

Integrations

See how to disable integrations in the integrations compatibility section.

Examples

dd.service.mapping

Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.service.mapping=postgresql:web-app-pg -jar path/to/application.jar

dd.tags

Setting a global env for spans and JMX metrics:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -jar path/to/application.jar

dd.trace.span.tags

Example with adding project:test to every span:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.trace.span.tags=project:test -jar path/to/application.jar

dd.trace.jmx.tags

Setting custom.type:2 on a JMX metric:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.trace.span.tags=project:test -Ddd.trace.jmx.tags=custom.type:2 -jar path/to/application.jar

dd.trace.methods

Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.trace.methods="hello.GreetingController[doSomeStuff,doSomeOtherStuff];hello.Randomizer[randomize]" -jar path/to/application.jar

dd.trace.db.client.split-by-instance

Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.env=dev -Ddd.service=web-app -Ddd.trace.db.client.split-by-instance=TRUE -jar path/to/application.jar

DB Instance 1, webappdb, now gets its own service name that is the same as the db.instance span metadata:

DB Instance 2, secondwebappdb, now gets its own service name that is the same as the db.instance span metadata:

Similarly on the service map, you would now see one web app making calls to two different Postgres databases.

dd.http.server.tag.query-string

Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.http.server.tag.query-string=TRUE -jar path/to/application.jar

dd.trace.enabled

Example with system property and debug app mode:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.trace.enabled=false -Ddatadog.slf4j.simpleLogger.defaultLogLevel=debug -jar path/to/application.jar

Debug app logs show that Tracing is disabled, not installing instrumentations.

dd.jmxfetch.config.dir and dd.jmxfetch.config

Example configuration:

• Either the combination of: DD_JMXFETCH_CONFIG_DIR=<DIRECTORY_PATH> + DD_JMXFETCH_CONFIG=conf.yaml
• Or directly: DD_JMXFETCH_CONFIG=<DIRECTORY_PATH>/conf.yaml

With the following content for conf.yaml:

init_config:
instances:
    - jvm_direct: true
      port: '<PORT>'
      conf:
          - include:
                bean:
                    - java.lang:type=MemoryPool,name=Metaspace
                attribute:
                    Usage.used:
                        metric_type: gauge
                        alias: sb.usage.used

Would produce the following result:

See the Java integration documentation to learn more about Java metrics collection with JMX fetch.

B3 Headers Extraction and Injection

Datadog APM tracer supports B3 headers extraction and injection for distributed tracing.

Distributed headers injection and extraction is controlled by configuring injection/extraction styles. Currently two styles are supported:

• Datadog: Datadog
• B3: B3

Injection styles can be configured using:

• System Property: -Ddd.propagation.style.inject=Datadog,B3
• Environment Variable: DD_PROPAGATION_STYLE_INJECT=Datadog,B3

The value of the property or environment variable is a comma (or space) separated list of header styles that are enabled for injection. By default only Datadog injection style is enabled.

Extraction styles can be configured using:

• System Property: -Ddd.propagation.style.extract=Datadog,B3
• Environment Variable: DD_PROPAGATION_STYLE_EXTRACT=Datadog,B3

The value of the property or environment variable is a comma (or space) separated list of header styles that are enabled for extraction. By default only Datadog extraction style is enabled.

If multiple extraction styles are enabled extraction attempt is done on the order those styles are configured and first successful extracted value is used.

Trace Reporting

To report a trace to Datadog the following happens:

• Trace completes
• Trace is pushed to an asynchronous queue of traces
  • Queue is size-bound and doesn’t grow past a set limit of 7000 traces
  • Once the size limit is reached, traces are discarded
  • A count of the total traces is captured to ensure accurate throughput
• In a separate reporting thread, the trace queue is flushed and traces are encoded via msgpack then sent to the Datadog Agent via http
• Queue flushing happens on a schedule of once per second

To see the actual code, documentation, and usage examples for any of the libraries and frameworks that Datadog supports, check the full list of auto-instrumented components for Java applications in the Integrations section.

Trace Annotation

Add the dd-trace-api dependency to your project. For Maven, add this to pom.xml:

<dependency>
    <groupId>com.datadoghq</groupId>
    <artifactId>dd-trace-api</artifactId>
    <version>{version}</version>
</dependency>

For Gradle, add:

implementation group: 'com.datadoghq', name: 'dd-trace-api', version: {version}

Now add @Trace to methods to have them be traced when running with dd-java-agent.jar. If the Agent is not attached, this annotation has no effect on your application.

@Trace annotations have the default operation name trace.annotation, while the method traced have the resource by default.

Performance

Java APM has minimal impact on the overhead of an application:

• No collections maintained by Java APM grow unbounded in memory
• Reporting traces does not block the application thread
• Java APM loads additional classes for trace collection and library instrumentation
• Java APM typically adds no more than a 3% increase in CPU usage
• Java APM typically adds no more than a 3% increase in JVM heap usage

Further Reading

Additional helpful documentation, links, and articles:

Datadog Java APM source codeGITHUB Explore your services, resources and tracesDOCUMENTATION