New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Tracing Java Applications

Installation and Getting Started

To begin tracing applications written in any language, first install and configure the Datadog Agent, see the additional documentation for tracing Docker applications or Kubernetes applications.

Next, download dd-java-agent.jar that contains the Agent class files:

wget -O dd-java-agent.jar 'https://repository.sonatype.org/service/local/artifact/maven/redirect?r=central-proxy&g=com.datadoghq&a=dd-java-agent&v=LATEST'

Finally, add the following JVM argument when starting your application in your IDE, Maven or Gradle application script, or java -jar command:

-javaagent:/path/to/the/dd-java-agent.jar

Note that dd-trace-java’s artifacts (dd-java-agent.jar, dd-trace-api.jar, dd-trace-ot.jar) support all JVM-based languages, i.e. Scala, Groovy, Kotlin, Clojure, etc. If you need support for a particular framework, consider making an open-source contribution.

Automatic Instrumentation

Automatic instrumentation for Java uses the java-agent instrumentation capabilities provided by the JVM. When a java-agent is registered, it has the ability to modify class files at load time. The java-agent uses the Byte Buddy framework to find the classes defined for instrumentation and modify those class bytes accordingly.

Instrumentation may come from auto-instrumentation, the OpenTracing api, or a mixture of both. Instrumentation generally captures the following info:

  • Timing duration is captured using the JVM’s nanotime clock unless a timestamp is provided from the OpenTracing api
  • Key/value tag pairs
  • Errors and stacktraces which are unhandled by the application
  • A total count of traces (requests) flowing through the system

Compatibility

Datadog officially supports the Java JRE 1.7 and higher of both Oracle JDK and OpenJDK. Datadog does not officially support any early-access versions of Java.

Integrations

Most integrations are enabled by default. The following setting can change the default to disabled.

  • System Property: -Ddd.integrations.enabled=false
  • Environment Variable: DD_INTEGRATIONS_ENABLED=false

Integrations can be enabled or disabled individually (overriding the default above).

  • System Property: -Ddd.integration.<integration-name>.enabled=true
  • Environment Variable: DD_INTEGRATION_<INTEGRATION_NAME>_ENABLED=true

(See below for each integration’s name.)

Beta integrations are disabled by default but can be enabled individually.

Web Framework Compatibility

dd-java-agent includes support for automatically tracing the following web frameworks.

ServerVersionsSupport TypeInstrumentation Names (used for configuration)
Akka-Http Server10.0+Fully Supportedakka-http, akka-http-server
Java Servlet Compatible2.3+, 3.0+Fully Supportedservlet, servlet-2, servlet-3
Jax-RS AnnotationsJSR311-APIFully Supportedjax-rs, jaxrs, jax-rs-annotations
Jetty (non-Servlet)8+Betajetty, jetty-8
Netty Http Server and Client4.0+Fully Supportednetty, netty-4.0, netty-4.1
Play2.4-2.6Fully Supportedplay
Ratpack1.4+Betaratpack
Spark Java2.3+Betasparkjava (requires jetty)
Spring Web (MVC)4.0+Fully Supportedspring-web
Spring WebFlux5.0+Fully Supportedspring-webflux
Vert.x-Web4.1.0+Fully Supported(requires netty)

Web Framework tracing provides: timing HTTP request to response, tags for the HTTP request (status code, method, etc), error and stacktrace capturing, linking work created within a web request and Distributed Tracing.

Note: Many application servers are Servlet compatible and are automatically covered by that instrumentation, such as Tomcat, Jetty, Websphere, Weblogic, etc. Also, frameworks like Spring Boot inherently work because it uses a Servlet compatible embedded application server.

Don’t see your desired web frameworks? Datadog is continually adding additional support. Contact Datadog support if you need help.

Networking Framework Compatibility

dd-java-agent includes support for automatically tracing the following networking frameworks.

FrameworkVersionsSupport TypeInstrumentation Names (used for configuration)
Apache HTTP Client4.0+Fully Supportedhttpclient
Apache HTTP Async Client4.0+Fully Supportedhttpasyncclient, apache-httpasyncclient
AWS Java SDK1.11+, 2.2+Fully Supportedaws-sdk
gRPC1.5+Fully Supportedgrpc, grpc-client, grpc-server
HttpURLConnectionallFully Supportedhttpurlconnection, urlconnection
Kafka-Clients0.11+Fully Supportedkafka
Kafka-Streams0.11+Fully Supportedkafka, kafka-streams
Jax RS Clients2.0+Fully Supportedjax-rs, jaxrs, jax-rs-client
JMS1 and 2Fully Supportedjms
Rabbit AMQP2.7+Fully Supportedamqp, rabbitmq
OkHTTP3.0+Fully Supportedokhttp, okhttp-3

Networking tracing provides: timing request to response, tags for the request (e.g. response code), error and stacktrace capturing, and distributed tracing.

Don’t see your desired networking framework? Datadog is continually adding additional support. Contact Datadog support if you need help.

Data Store Compatibility

dd-java-agent includes support for automatically tracing the following database frameworks/drivers.

DatabaseVersionsSupport TypeInstrumentation Names (used for configuration)
Couchbase2.0+Fully Supportedcouchbase
Cassandra3.XFully Supportedcassandra
Elasticsearch Transport2.0+Fully Supportedelasticsearch, elasticsearch-transport, elasticsearch-transport-{2,5,6} (pick one)
Elasticsearch Rest5.0+Fully Supportedelasticsearch, elasticsearch-rest, elasticsearch-rest-5, elasticsearch-rest-6
Hibernate3.5+Fully Supportedhibernate
JDBCN/AFully Supportedjdbc
Jedis1.4+Fully Supportedredis
Lettuce5.0+Fully Supportedlettuce
MongoDB3.0+Fully Supportedmongo
SpyMemcached2.12+Fully Supportedspymemcached

dd-java-agent is also compatible with common JDBC drivers including:

  • Apache Derby
  • Firebird SQL
  • H2 Database Engine
  • HSQLDB
  • IBM DB2
  • MariaDB
  • MSSQL (Microsoft SQL Server)
  • MySQL
  • Oracle
  • Postgres SQL

Datastore tracing provides: timing request to response, query info (e.g. a sanitized query string), and error and stacktrace capturing.

Don’t see your desired datastores? Datadog is continually adding additional support. Contact Datadog support if you need help.

Other Framework Compatibility

dd-java-agent includes support for automatically tracing the following other frameworks.

FrameworkVersionsSupport TypeInstrumentation Names (used for configuration)
Slf4J MDC1+Fully Supportedmdc (See also dd.logs.injection config)
JSP Rendering2.3+Fully Supportedjsp, jsp-render
Dropwizard Views0.7+Fully Supporteddropwizard, dropwizard-view
Hystrix1.4+Fully Supportedhystrix
Twilio SDK0+Fully Supportedtwilio-sdk

Don’t see your desired framework? Datadog is continually adding additional support. Contact Datadog support if you need help.

To improve visibility into applications using unsupported frameworks, consider:

  • Adding custom instrumentation (with OpenTracing or the @Trace annotation).
  • Submitting a pull request with instrumentation for inclusion in a future release.
  • Contacting Datadog support and submitting a feature request.

Configuration

The tracer is configured using System Properties and Environment Variables as follows: (See integration specific config in the integrations section above.)

System PropertyEnvironment VariableDefaultDescription
dd.trace.enabledDD_TRACE_ENABLEDtrueWhen false tracing agent is disabled.
dd.trace.configDD_TRACE_CONFIGnullOptional path to a file were configuration properties are provided one per each line. E.g. dd.trace.enabled=my_service-name
dd.service.nameDD_SERVICE_NAMEunnamed-java-appThe name of a set of processes that do the same job. Used for grouping stats for your application.
dd.service.mappingDD_SERVICE_MAPPINGnull(Example: mysql:my-service-name-db) Dynamically rename services via configuration. Useful for making databases have distinct names across different services.
dd.writer.typeDD_WRITER_TYPEDDAgentWriterDefault value sends traces to the Agent. Configuring with LoggingWriter instead writes traces out to the console.
dd.agent.hostDD_AGENT_HOSTlocalhostHostname for where to send traces to. If using a containerized environment, configure this to be the host IP. See Tracing Docker Applications for more details.
dd.trace.agent.portDD_TRACE_AGENT_PORT8126Port number the Agent is listening on for configured host.
dd.trace.global.tagsDD_TRACE_GLOBAL_TAGSnull(Example: key1:value1,key2:value2) A list of default tags to be added to every span and every JMX metric. This value is merged into trace.span.tags and trace.jmx.tags to provide single place to configure both.
dd.trace.span.tagsDD_TRACE_SPAN_TAGSnull(Example: key1:value1,key2:value2) A list of default tags to be added to every span. Tags of the same name added directly to a span overwrite the defaults provided here.
dd.trace.jmx.tagsDD_TRACE_JMX_TAGSnull(Example: key1:value1,key2:value2) A list of default tags to be added to every JMX metric. Tags of the same name added in JMX metrics configuration overwrite the defaults provided here.
dd.trace.header.tagsDD_TRACE_HEADER_TAGSnull(Example: CASE-insensitive-Header:my-tag-name,User-ID:userId) A map of header keys to tag names. Automatically apply header values as tags on traces.
dd.trace.annotationsDD_TRACE_ANNOTATIONS(listed here)(Example: com.some.Trace;io.other.Trace) A list of method annotations to treat as @Trace.
dd.trace.methodsDD_TRACE_METHODSnull(Example: package.ClassName[method1,method2,...];AnonymousClass$1[call]) List of class/interface and methods to trace. Similar to adding @Trace, but without changing code.
dd.trace.partial.flush.min.spansDD_TRACE_PARTIAL_FLUSH_MIN_SPANS1000Set a number of partial spans to flush on. Useful to reduce memory overhead when dealing with heavy traffic or long running traces.
dd.trace.report-hostnameDD_TRACE_REPORT_HOSTNAMEfalseWhen enabled, it adds the detected hostname to trace metadata
dd.trace.db.client.split-by-instanceDD_TRACE_DB_CLIENT_SPLIT_BY_INSTANCEfalseWhen set to true db spans get assigned the instance name as the service name
dd.http.client.tag.query-stringDD_HTTP_CLIENT_TAG_QUERY_STRINGfalseWhen set to true query string parameters and fragment get added to web client spans
dd.http.server.tag.query-stringDD_HTTP_SERVER_TAG_QUERY_STRINGfalseWhen set to true query string parameters and fragment get added to web server spans
dd.jmxfetch.enabledDD_JMXFETCH_ENABLEDtrueEnable collection of JMX metrics by Java Tracing Agent.
dd.jmxfetch.metrics-configsDD_JMXFETCH_METRICS_CONFIGSnull(Example: /file/loction1,/file/location2) Additional metrics configuration file for JMX metrics collection.
dd.jmxfetch.check-periodDD_JMXFETCH_CHECK_PERIOD1500How often to send JMX metrics (in ms).
dd.jmxfetch.refresh-beans-periodDD_JMXFETCH_REFRESH_BEANS_PERIOD600How often to refresh list of avalable JMX beans (in seconds).
dd.jmxfetch.statsd.hostDD_JMXFETCH_STATSD_HOSTsame as agent.hostStatsd host to send JMX metrics to.
dd.jmxfetch.statsd.portDD_JMXFETCH_STATSD_PORT8125Statsd port to send JMX metrics to.
dd.logs.injectionDD_LOGS_INJECTIONfalseEnabled automatic MDC key injection for Datadog trace and span ids. See Advanced Usage for details

Note:

  • If the same key type is set for both, the system property configuration takes priority.
  • System properties can be used as JVM parameters.

B3 Headers Extraction and Injection

Datadog APM tracer supports B3 headers extraction and injection for distributed tracing.

Distributed headers injection and extraction is controlled by configuring injection/extraction styles. Currently two styles are supported:

  • Datadog: Datadog
  • B3: B3

Injection styles can be configured using:

  • System Property: -Ddd.propagation.style.inject=Datadog,B3
  • Environment Variable: DD_PROPAGATION_STYLE_INJECTION=Datadog,B3

The value of the property or environment variable is a comma (or space) separated list of header styles that are enabled for injection. By default only Datadog injection style is enabled.

Extraction styles can be configured using:

  • System Property: -Ddd.propagation.style.extract=Datadog,B3
  • Environment Variable: DD_PROPAGATION_STYLE_EXTRACT=Datadog,B3

The value of the property or environment variable is a comma (or space) separated list of header styles that are enabled for extraction. By default only Datadog extraction style is enabled.

If multiple extraction styles are enabled extraction attempt is done on the order those styles are configured and first successful extracted value is used.

Trace Reporting

To report a trace to Datadog the following happens:

  • Trace completes
  • Trace is pushed to an asynchronous queue of traces
    • Queue is size-bound and doesn’t grow past a set limit of 7000 traces
    • Once the size limit is reached, traces are discarded
    • A count of the total traces is captured to ensure accurate throughput
  • In a separate reporting thread, the trace queue is flushed and traces are encoded via msgpack then sent to the Datadog Agent via http
  • Queue flushing happens on a schedule of once per second

To see the actual code, documentation, and usage examples for any of the libraries and frameworks that Datadog supports, check the full list of auto- instrumented components for Java applications in the Integrations section.

Trace Annotation

Add the dd-trace-api dependency to your project. For Maven, add this to pom.xml:

<dependency>
    <groupId>com.datadoghq</groupId>
    <artifactId>dd-trace-api</artifactId>
    <version>{version}</version>
</dependency>

For Gradle, add:

compile group: 'com.datadoghq', name: 'dd-trace-api', version: {version}

Now add @Trace to methods to have them be traced when running with dd-java-agent.jar. If the Agent is not attached, this annotation has no effect on your application.

Performance

Java APM has minimal impact on the overhead of an application:

  • No collections maintained by Java APM grow unbounded in memory
  • Reporting traces does not block the application thread
  • Java APM typically adds no more than a 3% increase in CPU usage
  • Java APM typically adds no more than a 3% increase in JVM heap usage

Change Agent Hostname

Configure your application level tracers to submit traces to a custom Agent hostname:

The Java Tracing Module automatically looks for and initializes with the ENV variables DD_AGENT_HOST and DD_TRACE_AGENT_PORT.

java -javaagent:<DD-JAVA-AGENT-PATH>.jar -jar <YOUR_APPLICATION_PATH>.jar

You can also use system properties:

java -javaagent:<DD-JAVA-AGENT-PATH>.jar \
     -Ddd.agent.host=$DD_AGENT_HOST \
     -Ddd.agent.port=$DD_TRACE_AGENT_PORT \
     -jar <YOUR_APPLICATION_PATH>.jar

Further Reading