The Service Map for APM is here!

Java log collection

Java logs are quite complex to handle, mainly because of stack traces. These stack traces are split into multiple lines which makes them difficult to associate to the original log event:

//4 events generated when only one is expected!
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

By asking your logging library to log into JSON, you will:

  • Ensure to have a stack_trace properly wrapped into the proper LogEvent
  • Ensure that all the attributes of a log event are properly extracted (severity, logger name, thread name, etc…)
  • You’ll have access to MDC, which are attributes you can attach to any log events

To send your logs to Datadog, we recommend logging to a file and then tailing that file with your Datadog Agent.

We also strongly encourage you to setup your logging libraries to produce your logs in JSON format to avoid sustaning custom parsing rules.

Here are setup examples for the log4j, slf4j and log4j2 logging libraries:

Configure your logger

Raw format

Add a new file appender to log4j.xml:

<appender name="fileAppender" class="org.apache.log4j.FileAppender">
  <param name="File" value="/logs/log4j.log" />
  <param name="Append" value="true" />
  <layout class="org.apache.log4j.PatternLayout">
      <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n" />
  </layout>
</appender>

Edit your log4j2.xml file:

 <File name="MyFile" fileName="logs/app.log" immediateFlush="true">
        <PatternLayout pattern="%d{yyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
 </File>
 <Loggers>
        <Root level="debug">
        <AppenderRef ref="MyFile" />
        </Root>
</Loggers>

Edit your logback.xml file:

<configuration>
(....)
   <timestamp key="byDay" datePattern="yyyyMMdd'T'HHmmss"/>

   <appender name="FILE" class="ch.qos.logback.core.FileAppender">
      <file> ~/logs/log-${byDay}.log </file>
      <append>true</append>
      <encoder>
          <Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</Pattern>
        </encoder>
   </appender>
(....)
    <root level="debug">
        <appender-ref ref="FILE" />
    </root>
</configuration>

JSON Format

It can be difficult to log in JSON with log4j. Because of this, we advise you to use a slf4j ship with a module called log4j-over-slf4j and then use logback for the json format.

To use log4j-over-slf4j in your own application, the first step is to locate and then replace log4j.jar with log4j-over-slf4j.jar. Note that you still need an slf4j binding and its dependencies for log4j-over-slf4j to work properly.

In most situations, replacing a jar file is all it takes in order to migrate from log4j to SLF4J. Edit your pom.xml file:

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>log4j-over-slf4j</artifactId>
  <version>1.7.13</version>
</dependency>

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>4.5.1</version>
</dependency>

<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-classic</artifactId>
  <version>1.1.3</version>
</dependency>

Once that done, edit your logback.xml file as described in the below Slf4j section.

There is a default log4j2 JSON Layout that can be used as shown in this example.

The JSON library we recommend for Logback is logstash-logback-encoder. One advantage is: it’s inside the main Maven repository.

To add it into your classpath, simply add the following dependency (version 4.5.1 on the example) in your pom.xml file:

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>4.5.1</version>
</dependency>

<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-classic</artifactId>
  <version>1.1.3</version>
</dependency>

Then edit your logback.xml file and update the encoder:

    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>logs/app.log</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <customFields>{"env":"prod"}</customFields>
        </encoder>
    </appender>

Configure the Datadog Agent

Create a file java.yaml in the Agent’s conf.d/ directory with the following content:

#Log section
logs:

    ## - type : file (mandatory) type of log input source (tcp / udp / file)
    ##   port / path : (mandatory) Set port if type is tcp or udp. Set path if type is file
    ##   service : (mandatory) name of the service owning the log
    ##   source : (mandatory) attribute that defines which integration is sending the logs
    ##   sourcecategory : (optional) Multiple value attribute. Can be used to refine the source attribute
    ##   tags: (optional) add tags to each logs collected

  - type: file
    path: /path/to/your/java/log.log
    service: java
    source: java
    sourcecategory: sourcecode
    # For multiline logs, if they start by the date with the format yyyy-mm-dd uncomment the following processing rule
    #log_processing_rules:
    #  - type: multi_line
    #    name: new_log_start_with_date
    #    pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])

Agentless logging

It is possible to stream logs from your application to Datadog or to the Datadog Agent directly. This is not the recommended setup, as handling connection issues should not be done directly in your application, but it might not be possible to log to a file when your application is running on a machine that cannot be accessed.

There are two steps to configure the Java application to stream logs directly to Datadog:

  1. Add the Logback logging library to your code (or build a bridge from your current logger to it)
  2. Configure it to send logs to Datadog

Bridge from Java logging libraries to Logback

This logging library can be linked from the most common Java ones:

Logging to a remote server in JSON may be difficult with Log4j. It is recommended that you use a SLF4J ship with a module called log4j-over-slf4j and then use Logback for the JSON format.

To use log4j-over-slf4j in your own application, the first step is to find-and-replace log4j.jar with log4j-over-slf4j.jar. In most situations, replacing a JAR file is all it takes in order to migrate from Log4j to SLF4J.

Then, edit the pom.xml file with the following content:

<dependency>
	<groupId>org.slf4j</groupId>
	<artifactId>log4j-over-slf4j</artifactId>
	<version>1.7.13</version>
</dependency>

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>4.5.1</version>
</dependency>

<dependency>
	<groupId>ch.qos.logback</groupId>
	<artifactId>logback-classic</artifactId>
	<version>1.1.3</version>
</dependency>

note: As a result of this migration, Log4j configuration files will no longer be picked up. Migrate your log4j.properties file to logback.xml with the Log4j translator.

Log4j2 allows logging to a remote host, but it does not offer the ability to prefix the logs by an API key. Because of this, it is recommended that you use a SLF4J ship with a module called log4j-over-slf4j and then use Logback for the JSON format.

To use log4j-over-slf4j in your own application, the first step is to find-and-replace log4j.jar with log4j-to-slf4j-2.11.jar.

Then, edit the pom.xml file with the following content:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-to-slf4j</artifactId>
    <version>2.11.0</version>
</dependency>

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>4.5.1</version>
</dependency>

<dependency>
	<groupId>ch.qos.logback</groupId>
	<artifactId>logback-classic</artifactId>
	<version>1.1.3</version>
</dependency>

notes:

To add Logback logstash-logback-encoder into your classpath, add the following dependency (version 4.5.1 on the example) in your pom.xml file:

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>4.5.1</version>
</dependency>

<dependency>
	<groupId>ch.qos.logback</groupId>
	<artifactId>logback-classic</artifactId>
	<version>1.1.3</version>
</dependency>

Logback configuration

Configure the Logback logger to stream logs directly to Datadog by adding the following in your logback.xml file:

<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
	<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>

<appender name="JSON_TCP" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
	<remoteHost>intake.logs.datadoghq.com</remoteHost>
  <port>10514</port>
  <prefix class="ch.qos.logback.classic.PatternLayout">
      <pattern><APIKEY> %mdc{keyThatDoesNotExist}</pattern>
  </prefix>
  <keepAliveDuration>1 minute</keepAliveDuration>
	<encoder class="net.logstash.logback.encoder.LogstashEncoder">
	</encoder>
</appender>
 
<root level="debug">
	<appender-ref ref="JSON_TCP" />
	<appender-ref ref="JSON" />
</root>

Notes:

  • Replace <API_KEY> with your Datadog API key value
  • %mdc{keyThatDoesNotExist} is added because the XML configuration trims whitespace, as explained here
  • See the list of available endpoints for the EU region

Getting further

Enrich your log events with contextual attributes.

Using the Key/Value parser

The Key/Value parser extracts any <key>=<value> pattern recognized in any log event.

To enrich your log events in Java, you can re-write messages in your code and introduce <key>=<value> sequences.

For instance if you have:

logger.info("Emitted 1001 messages during the last 93 seconds for customer scope prod30");

You can change it to:

logger.info("Emitted quantity=1001 messages during the last durationInMs=93180 ms for customer scope=prod30");

With the Key/Value parser enabled, Datadog automatically extracts each pair from your final JSON document:

{
    //...
    "message" : "Emitted quantity=1001 messages during the last durationInMs=93180 ms for customer scope=prod30",
    "scope" : "prod30",
    "durationInMs" : 93180,
    "quantity" : 1001
    //...
}

So you can exploit scope as a field, and durationInMs and quantity as log measures.

MDC (Mapped Diagnostic Context)

Another option to enrich your logs is to use Java’s MDC (Mapped Diagnostic Contexts).

If you use Logback, use the following Java code:

...
MDC.put("scope", "prod30");
logger.info("Emitted 1001 messages during the last 93 seconds");
...

To generate this final JSON document:

{
    "message" : "Emitted 1001 messages during the last 93 seconds",
    "scope" : "prod30",
}

MDC are great but for some reason only string types are allowed. Therefore, providing numerical values for metrics with MDCs would be a bad idea

Further Reading