Tracing Java Applications
New announcements from Dash: Incident Management, Continuous Profiler, and more! New announcements from Dash!

Tracing Java Applications

Compatibility requirements

The Java Tracer requires Java JRE 1.7 and higher for either Oracle JDK and OpenJDK. Datadog does not officially support any early-access versions of Java. For a full list of supported libraries, visit the Compatibility Requirements page.

Installation and Getting Started

Follow the Quickstart instructions within the Datadog app for the best experience, including:

  • Step-by-step instructions scoped to your deployment configuration (hosts, Docker, Kubernetes, or Amazon ECS).
  • Dynamically set service, env and version tags.
  • Enable the Continuous Profiler, App Analytics, and Trace ID injection into logs during setup.

Otherwise, to begin tracing applications written in any language:

  1. Install and configure the Datadog Agent, see the additional documentation for tracing Docker applications or Kubernetes applications.

  2. Download dd-java-agent.jar that contains the Agent class files:

    wget -O dd-java-agent.jar
  3. Add the following JVM argument when starting your application in your IDE, Maven or Gradle application script, or java -jar command:

  4. Add Configuration options for tracing and ensure you are setting environment variables or passing system properties as JVM arguments, particularly for service, environment, logs injection, profiling, and optionally runtime metrics—all the metrics you intend to use. See the examples below. Note that using the in-app Quickstart instructions generates these for you.


  • Use the documentation for your IDE to figure out the right way to pass in -javaagent and other JVM arguments. Here are instructions for some commonly used frameworks:

    In the administrative console:

    1. Select Servers. Under Server Type, select WebSphere application servers and select your server.
    2. Select Java and Process Management > Process Definition.
    3. In the Additional Properties section, click Java Virtual Machine.
    4. In the Generic JVM arguments text field, enter:


    For additional details and options, see the WebSphere docs.

    If your app is called my_app.jar, create a my_app.conf, containing:


    For more information, see the Spring Boot documentation.

    Open your Tomcat startup script file, for example, and add:

    CATALINA_OPTS="$CATALINA_OPTS -javaagent:/path/to/dd-java-agent.jar"

    Or on Windows, catalina.bat:

    set CATALINA_OPTS_OPTS=%CATALINA_OPTS_OPTS% -javaagent:"c:\path\to\dd-java-agent.jar"

    Add the following line to the end of

    JAVA_OPTS="$JAVA_OPTS -javaagent:/path/to/dd-java-agent.jar"

    On Windows, add the following line to the end of standalone.conf.bat:

    set "JAVA_OPTS=%JAVA_OPTS% -javaagent:X:/path/to/dd-java-agent.jar"

    For more details, see the JBoss documentation.

    If you use to start Jetty as a service, edit it to add:

    JAVA_OPTIONS="${JAVA_OPTIONS} -javaagent:/path/to/dd-java-agent.jar"

    If you use start.ini to start Jetty, add the following line (under --exec, or add --exec line if it isn’t there yet):

  • If you’re adding the -javaagent argument to your java -jar command, it needs to be added before the -jar argument, that is as a JVM option, not as an application argument. For example:

    java -javaagent:/path/to/dd-java-agent.jar -jar my_app.jar

    For more information, see the Oracle documentation.

  • dd-trace-java’s artifacts (dd-java-agent.jar, dd-trace-api.jar, dd-trace-ot.jar) support all JVM-based languages, i.e. Scala, Groovy, Kotlin, Clojure, etc. If you need support for a particular framework, consider making an open-source contribution.

Automatic Instrumentation

Automatic instrumentation for Java uses the java-agent instrumentation capabilities provided by the JVM. When a java-agent is registered, it has the ability to modify class files at load time. The java-agent uses the Byte Buddy framework to find the classes defined for instrumentation and modify those class bytes accordingly.

Instrumentation may come from auto-instrumentation, the OpenTracing api, or a mixture of both. Instrumentation generally captures the following info:

  • Timing duration is captured using the JVM’s nanotime clock unless a timestamp is provided from the OpenTracing API
  • Key/value tag pairs
  • Errors and stacktraces which are unhandled by the application
  • A total count of traces (requests) flowing through the system


All configuration options below have system property and environment variable equivalents. If the same key type is set for both, the system property configuration takes priority. System properties can be set as JVM flags.

System PropertyEnvironment VariableDefaultDescription
dd.serviceDD_SERVICEunnamed-java-appThe name of a set of processes that do the same job. Used for grouping stats for your application. Available for versions 0.50.1+.
dd.tagsDD_TAGSnull(Example: layer:api,team:intake) A list of default tags to be added to every span, profile, and JMX metric. If DD_ENV or DD_VERSION is used, it will override any env or version tag defined in DD_TAGS. Available for versions 0.50.1+.
dd.envDD_ENVnoneYour application environment (e.g. production, staging, etc.). Available for versions 0.48+.
dd.versionDD_VERSIONnullYour application version (e.g. 2.5, 202003181415, 1.3-alpha, etc.). Available for versions 0.48+.
dd.logs.injectionDD_LOGS_INJECTIONfalseEnabled automatic MDC key injection for Datadog trace and span IDs. See Advanced Usage for details.
dd.trace.configDD_TRACE_CONFIGnullOptional path to a file where configuration properties are provided one per each line. For instance, the file path can be provided as via -Ddd.trace.config=<FILE_PATH>.properties, with setting the service name in the file with dd.service=<SERVICE_NAME>
dd.service.mappingDD_SERVICE_MAPPINGnull(Example: mysql:my-mysql-service-name-db, postgres:my-postgres-service-name-db) Dynamically rename services via configuration. Useful for making databases have distinct names across different services.
dd.writer.typeDD_WRITER_TYPEDDAgentWriterDefault value sends traces to the Agent. Configuring with LoggingWriter instead writes traces out to the console.
dd.agent.hostDD_AGENT_HOSTlocalhostHostname for where to send traces to. If using a containerized environment, configure this to be the host IP. See Tracing Docker Applications for more details.
dd.trace.agent.portDD_TRACE_AGENT_PORT8126Port number the Agent is listening on for configured host.
dd.trace.agent.unix.domain.socketDD_TRACE_AGENT_UNIX_DOMAIN_SOCKETnullThis can be used to direct trace traffic to a proxy, to later be sent to a remote Datadog Agent.
dd.trace.agent.timeoutDD_TRACE_AGENT_TIMEOUT10Timeout in seconds for network interactions with the Datadog Agent.
dd.trace.header.tagsDD_TRACE_HEADER_TAGSnull(Example: CASE-insensitive-Header:my-tag-name,User-ID:userId) A map of header keys to tag names. Automatically apply header values as tags on traces.
dd.trace.annotationsDD_TRACE_ANNOTATIONS(listed here)(Example: com.some.Trace;io.other.Trace) A list of method annotations to treat as @Trace.
dd.trace.methodsDD_TRACE_METHODSnull(Example: "package.ClassName[method1,method2,...];AnonymousClass$1[call]") List of class/interface and methods to trace. Similar to adding @Trace, but without changing code.
dd.trace.partial.flush.min.spansDD_TRACE_PARTIAL_FLUSH_MIN_SPANS1000Set a number of partial spans to flush on. Useful to reduce memory overhead when dealing with heavy traffic or long running traces.
dd.trace.split-by-tagsDD_TRACE_SPLIT_BY_TAGSnull(Example: aws.service) Used to rename spans to be identified with the corresponding service tag
dd.trace.db.client.split-by-instanceDD_TRACE_DB_CLIENT_SPLIT_BY_INSTANCEfalseWhen set to true db spans get assigned the instance name as the service name set to true sends tracer health metrics as dd.jmxfetch.statsd.hostStatsd host to send health metrics to as dd.jmxfetch.statsd.portStatsd port to send health metrics to
dd.http.client.tag.query-stringDD_HTTP_CLIENT_TAG_QUERY_STRINGfalseWhen set to true query string parameters and fragment get added to web client spans
dd.http.client.error.statusesDD_HTTP_CLIENT_ERROR_STATUSES400-499A range of errors can be accepted. By default 4xx errors are reported as errors for http clients. This configuration overrides that. Ex. dd.http.client.error.statuses=400-499
dd.http.server.error.statusesDD_HTTP_SERVER_ERROR_STATUSES500-599A range of errors can be accepted. By default 5xx status codes are reported as errors for http servers. This configuration overrides that. Ex. dd.http.server.error.statuses=500-599
dd.http.server.tag.query-stringDD_HTTP_SERVER_TAG_QUERY_STRINGfalseWhen set to true query string parameters and fragment get added to web server spans
dd.trace.enabledDD_TRACE_ENABLEDtrueWhen false tracing agent is disabled.
dd.jmxfetch.enabledDD_JMXFETCH_ENABLEDtrueEnable collection of JMX metrics by Java Tracing Agent.
dd.jmxfetch.config.dirDD_JMXFETCH_CONFIG_DIRnull(Example: /opt/datadog-agent/etc/conf.d) Additional configuration directory for JMX metrics collection. The Java Agent looks for jvm_direct:true in the instance section in the yaml file to change configuration.
dd.jmxfetch.configDD_JMXFETCH_CONFIGnull(Example: activemq.d/conf.yaml,jmx.d/conf.yaml) Additional metrics configuration file for JMX metrics collection. The Java Agent looks for jvm_direct:true in the instance section in the yaml file to change configuration.
dd.jmxfetch.check-periodDD_JMXFETCH_CHECK_PERIOD1500How often to send JMX metrics (in ms).
dd.jmxfetch.refresh-beans-periodDD_JMXFETCH_REFRESH_BEANS_PERIOD600How often to refresh list of avalable JMX beans (in seconds).
dd.jmxfetch.statsd.hostDD_JMXFETCH_STATSD_HOSTsame as agent.hostStatsd host to send JMX metrics to. If you are using Unix Domain Sockets, use an argument like ‘unix://PATH_TO_UDS_SOCKET’. Example: unix:///var/datadog-agent/dsd.socket
dd.jmxfetch.statsd.portDD_JMXFETCH_STATSD_PORT8125StatsD port to send JMX metrics to. If you are using Unix Domain Sockets, input 0.
dd.integration.opentracing.enabledDD_INTEGRATION_OPENTRACING_ENABLEDtrueBy default the tracing client detects if a GlobalTracer is being loaded and dynamically registers a tracer into it. By turning this to false, this removes any tracer dependency on OpenTracing.
dd.hystrix.tags.enabledDD_HYSTRIX_TAGS_ENABLEDFalseBy default the Hystrix group, command, and circuit state tags are not enabled. This property enables them.
dd.trace.servlet.async-timeout.errorDD_TRACE_SERVLET_ASYNC_TIMEOUT_ERRORTrueBy default, long running asynchronous requests will be marked as an error, setting this value to false allows to mark all timeouts as successful requests.



See how to disable integrations in the integrations compatability section.



Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.service.mapping=postgresql:web-app-pg -jar path/to/application.jar


Setting a global env for spans and JMX metrics:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -jar path/to/application.jar


Example with adding project:test to every span:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.trace.span.tags=project:test -jar path/to/application.jar


Setting custom.type:2 on a JMX metric:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.trace.span.tags=project:test -Ddd.trace.jmx.tags=custom.type:2 -jar path/to/application.jar


Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.trace.methods="hello.GreetingController[doSomeStuff,doSomeOtherStuff];hello.Randomizer[randomize]" -jar path/to/application.jar


Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.env=dev -Ddd.service=web-app -Ddd.trace.db.client.split-by-instance=TRUE -jar path/to/application.jar

DB Instance 1, webappdb, now gets its own service name that is the same as the db.instance span metadata:

DB Instance 2, secondwebappdb, now gets its own service name that is the same as the db.instance span metadata:

Similarly on the service map, you would now see one web app making calls to two different Postgres databases.


Example with system property:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.service=web-app -Ddd.env=dev -Ddd.http.server.tag.query-string=TRUE -jar path/to/application.jar


Example with system property and debug app mode:

java -javaagent:/path/to/dd-java-agent.jar -Ddd.trace.enabled=false -Ddatadog.slf4j.simpleLogger.defaultLogLevel=debug -jar path/to/application.jar

Debug app logs show that Tracing is disabled, not installing instrumentations.

dd.jmxfetch.config.dir and dd.jmxfetch.config

Example configuration:

  • Either the combination of: DD_JMXFETCH_CONFIG_DIR=<DIRECTORY_PATH> + DD_JMXFETCH_CONFIG=conf.yaml
  • Or directly: DD_JMXFETCH_CONFIG=<DIRECTORY_PATH>/conf.yaml

With the following content for conf.yaml:

    - jvm_direct: true
      port: '<PORT>'
          - include:
                    - java.lang:type=MemoryPool,name=Metaspace
                        metric_type: gauge
                        alias: sb.usage.used

Would produce the following result:

See the Java integration documentation to learn more about Java metrics collection with JMX fetch.

B3 Headers Extraction and Injection

Datadog APM tracer supports B3 headers extraction and injection for distributed tracing.

Distributed headers injection and extraction is controlled by configuring injection/extraction styles. Currently two styles are supported:

  • Datadog: Datadog
  • B3: B3

Injection styles can be configured using:

  • System Property:,B3
  • Environment Variable: DD_PROPAGATION_STYLE_INJECT=Datadog,B3

The value of the property or environment variable is a comma (or space) separated list of header styles that are enabled for injection. By default only Datadog injection style is enabled.

Extraction styles can be configured using:

  • System Property:,B3
  • Environment Variable: DD_PROPAGATION_STYLE_EXTRACT=Datadog,B3

The value of the property or environment variable is a comma (or space) separated list of header styles that are enabled for extraction. By default only Datadog extraction style is enabled.

If multiple extraction styles are enabled extraction attempt is done on the order those styles are configured and first successful extracted value is used.

Trace Reporting

To report a trace to Datadog the following happens:

  • Trace completes
  • Trace is pushed to an asynchronous queue of traces
    • Queue is size-bound and doesn’t grow past a set limit of 7000 traces
    • Once the size limit is reached, traces are discarded
    • A count of the total traces is captured to ensure accurate throughput
  • In a separate reporting thread, the trace queue is flushed and traces are encoded via msgpack then sent to the Datadog Agent via http
  • Queue flushing happens on a schedule of once per second

To see the actual code, documentation, and usage examples for any of the libraries and frameworks that Datadog supports, check the full list of auto-instrumented components for Java applications in the Integrations section.

Trace Annotation

Add the dd-trace-api dependency to your project. For Maven, add this to pom.xml:


For Gradle, add:

compile group: 'com.datadoghq', name: 'dd-trace-api', version: {version}

Now add @Trace to methods to have them be traced when running with dd-java-agent.jar. If the Agent is not attached, this annotation has no effect on your application.

@Trace annotations have the default operation name trace.annotation, while the method traced have the resource by default.


Java APM has minimal impact on the overhead of an application:

  • No collections maintained by Java APM grow unbounded in memory
  • Reporting traces does not block the application thread
  • Java APM loads additional classes for trace collection and library instrumentation
  • Java APM typically adds no more than a 3% increase in CPU usage
  • Java APM typically adds no more than a 3% increase in JVM heap usage

Change Agent Hostname

Configure your application level tracers to submit traces to a custom Agent hostname:

The Java Tracing Module automatically looks for and initializes with the ENV variables DD_AGENT_HOST and DD_TRACE_AGENT_PORT.

java -javaagent:<DD-JAVA-AGENT-PATH>.jar -jar <YOUR_APPLICATION_PATH>.jar

You can also use system properties:

java -javaagent:<DD-JAVA-AGENT-PATH>.jar \$DD_AGENT_HOST \
     -Ddd.agent.port=$DD_TRACE_AGENT_PORT \

Further Reading