Datadog Synthetics is now available!

Java

Library Library

Overview

The JMX integration collects metrics from applications that expose JMX metrics.

A lightweight Java plugin named JMXFetch is called by the Datadog Agent to connect to the MBean Server and to collect these metrics. This plugin sends metrics to the Datadog Agent using the DogStatsD server running within the Agent. This functionality is also leveraged in the integrations for ActiveMQ, Cassandra, Solr, and Tomcat. JMXFetch also sends service checks that report on the status of your monitored instances.

JMXFetch is only compatible with Java >= 1.7.

JMX checks have a limit of 350 metrics per instance.

If you are running JMX within Docker, refer to the dedicated Docker-JMX documentation.

Setup

Installation

Make sure you can open a JMX remote connection.

A remote connection is required for the Datadog Agent to connect to the JVM, even when the two are on the same host.

Configuration

  1. Configure the Agent to connect using JMX and edit it according to your needs. Here is a sample jmx.d/conf.yaml file:
init_config:
  custom_jar_paths: # optional
    - <CUSTOM_JAR_FILE_PATH>.jar
  #is_jmx: true

instances:
  - host: localhost
    port: 7199
    user: <USER_NAME>
    password: <PASSWORD>

    jmx_url: "service:jmx:rmi:///jndi/rmi://myhost.host:9999/<CUSTOM_PATH>" # optional

    name: jmx_instance  # optional
    java_bin_path: <JAVA_PATH>
    java_options: "-Xmx200m -Xms50m"
    trust_store_path: <TRUST_STORE_PATH>.jks
    trust_store_password: <PASSWORD>

    process_name_regex: .*<PROCESS_NAME>.*
    tools_jar_path: /usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar
    refresh_beans: 600 # optional (in seconds)
    tags:
      env: stage
      <TAG_KEY>:<TAG_VALUE>

    conf:
      - include:
          domain: <DOMAIN_NAME_1>
          tags:
              simple: $attr0
              raw_value: <CHOOSEN_VALUE>
              multiple: $attr0-$attr1
          bean:
            - <BEAN_NAME_1>
            - <BEAN_NAME_2>
          attribute:
            attribute1:
              metric_type: counter
              alias: jmx.<METRIC_NAME_ATTRIBUTE_1>
            attribute2:
              metric_type: gauge
              alias: jmx.<METRIC_NAME_ATTRIBUTE_2>

      - include:
          domain: <DOMAIN_NAME_2>
        exclude:
          bean:
            - <EXCLUDED_BEAN_NAME>
      - include:
          domain_regex: <DOMAIN_REGEX>
        exclude:
          bean_regex:
            - <EXCLUDED_BEAN_NAME_REGEX>
      - include:
          bean_regex: regex_topic=(.*?)
          attribute: 
            atteibute1:
              metric_type: gauge
              alias: jmx.<ATTRIBUTE_NAME_WITH_REGEX_TAG>

          ## The following submits jmx.<ATTRIBUTE_NAME_WITH_REGEX_TAG> bean with tags:
          ## `hostregex:<beanParameter>`
          ## `typeregex:<beanParameter>`
          ## `contextregex<beanParameter>`
          ## `optional:tag`
          tags:
              TypeRegex: $1
              HostRegex: $2
              contextRegex: $3
              optional: tag

Note: To run more than one JMX check, create configuration files with the format jmx_<INDEX>.d/conf.yaml (e.g. jmx_1.d/conf.yaml, jmx_2.d/conf.yaml, etc). Each folder should be stored in the conf.d directory. Include the is_jmx option set to true in those configuration files.

Configuration Options

Option Required Description
custom_jar_paths No Allows specifying custom jars that are added to the classpath of the Agent’s JVM.
jmx_url No If the Agent needs to connect to a non-default JMX URL, specify it here instead of a host and port. If you use this you need to specify a name for the instance.
is_jmx No Allows creating different configuration files for each application rather than using a single long JMX file. Include the option in each configuration file as explained in the note from the Configuration section.
collect_default_metrics No Each integration contains a metrics.yaml file that contains a list of default beans to collect. Setting this to True automatically collects those metrics without explicitly adding them to the yaml file. This is typically used for setting up the configuration with Autodiscovery to reduce the size of the configuration object.
name No Used in conjunction with jmx_url.
java_bin_path No Should be set if the Agent cannot find your Java executable.
java_options No Java JVM options
trust_store_path and trust_store_password No Should be set if SSL is enabled.
process_name_regex No Instead of specifying a host and port or jmx_url, the Agent can connect using the attach API. This requires the JDK to be installed and the path to tools.jar to be set.
tools_jar_path No To be set when process_name_regex is set.
refresh_beans No Refresh period for refreshing the matching MBeans list. Default is 600 seconds. Decreasing this value may result in increased CPU usage.

The conf parameter is a list of dictionaries. Only 2 keys are allowed in this dictionary:

Key Required Description
include Yes A dictionary of filters - any attribute that matches these filters are collected unless it also matches the “exclude” filters (see below).
exclude No A dictionary of filters - attributes that match these filters are not collected.

Tags are automatically added to metrics based on the actual MBean name. You can explicitly specify supplementary tags. For instance, assuming the following MBean is exposed by your monitored application:

mydomain:attr0=val0,attr1=val1

It would create a metric called mydomain (or some variation depending on the attribute inside the bean) with tags: attr0:val0, attr1:val1, domain:mydomain, simple:val0, raw_value:my_chosen_value, multiple:val0-val1.

If you specify an alias in an include key that is formatted as camel case, it is converted to snake case. For example, MyMetricName is shown in Datadog as my_metric_name.

Description of the filters

Each include or exclude dictionary supports the following keys:

Key Description
domain A list of domain names (e.g. java.lang).
domain_regex A list of regexes on the domain name (e.g. java\.lang.*).
bean or bean_name A list of full bean names (e.g. java.lang:type=Compilation).
bean_regex A list of regexes on the full bean names (e.g. java\.lang.*[,:]type=Compilation.*). You can use capture groups in your regex to supply as tag values. See example configuration above.
attribute A list or a dictionary of attribute names (see below for more details).

The regexes defined in domain_regex and bean_regex must conform to Java’s regular expression format.

The domain_regex and bean_regex filters were added in version 5.5.0.

On top of these parameters, the filters support “custom” keys which allows you to filter by bean parameters. For example, if you want to collect metrics regarding the Cassandra cache, you could use the type: - Caches filter:

conf:
- include:
    domain: org.apache.cassandra.db
    type:
      - Caches

The Attribute filter

The attribute filter can accept two types of values:

  • A dictionary whose keys are attributes names:
  conf:
    - include:
        attribute:
          maxThreads:
            alias: tomcat.threads.max
            metric_type: gauge
          currentThreadCount:
            alias: tomcat.threads.count
            metric_type: gauge
          bytesReceived:
            alias: tomcat.bytes_rcvd
            metric_type: counter

In that case you can specify an alias for the metric that becomes the metric name in Datadog. You can also specify the metric type either a gauge or a counter. If you choose counter, a rate per second is computed for this metric.

  • A list of attributes names:
  conf:
    - include:
        domain: org.apache.cassandra.db
        attribute:
          - BloomFilterDiskSpaceUsed
          - BloomFilterFalsePositives
          - BloomFilterFalseRatio
          - Capacity
          - CompressionRatio
          - CompletedTasks
          - ExceptionCount
          - Hits
          - RecentHitRate

In that case:

* The metric type is a gauge.
* The metric name is `jmx.<DOMAIN_NAME>.<ATTRIBUTE_NAME>`.

Here is another filtering example:

instances:
  - host: 127.0.0.1
    name: jmx_instance
    port: 9999

init_config:
  conf:
    - include:
        bean: org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency
        attribute:
          - OneMinuteRate
          - 75thPercentile
          - 95thPercentile
          - 99thPercentile

Older versions

List of filters is only supported in Datadog Agent > 5.3.0. If you are using an older version, use singletons and multiple include statements instead.

    # Datadog Agent > 5.3.0
      conf:
        - include:
            domain: domain_name
            bean:
              - first_bean_name
              - second_bean_name
    ...


    # Older Datadog Agent versions
      conf:
        - include:
            domain: domain_name
            bean: first_bean_name
        - include:
            domain: domain_name
            bean: second_bean_name
    ...

Validation

Run the Agent’s status subcommand and look for your JMX check under the JMXFetch section.

Additionally, JMX checks have a default configuration that collect 11 metrics from your JMX application. Check the Metrics Explorer for: jvm.heap_memory, jvm.non_heap_memory, or jvm.gc.cms.count.

Data Collected

Metrics

jvm.heap_memory
(gauge)
The total Java heap memory used.
shown as byte
jvm.heap_memory_committed
(gauge)
The total Java heap memory committed to be used.
shown as byte
jvm.heap_memory_init
(gauge)
The initial Java heap memory allocated.
shown as byte
jvm.heap_memory_max
(gauge)
The maximum Java heap memory available.
shown as byte
jvm.non_heap_memory
(gauge)
The total Java non-heap memory used.
shown as byte
jvm.non_heap_memory_committed
(gauge)
The total Java non-heap memory committed to be used.
shown as byte
jvm.non_heap_memory_init
(gauge)
The initial Java non-heap memory allocated.
shown as byte
jvm.non_heap_memory_max
(gauge)
The maximum Java non-heap memory available.
shown as byte
jvm.thread_count
(gauge)
The number of live threads.
shown as thread
jvm.gc.cms.count
(gauge)
The total number of garbage collections that have occurred.
jvm.gc.parnew.time
(gauge)
The approximate accumulated garbage collection time elapsed.
shown as millisecond

Troubleshooting

Consult the list of JMX troubleshooting commands.

The 350 metric limit

Due to the nature of these integrations, it is possible to submit an extremely high number of metrics directly to Datadog. Many customers agree that some of these metrics are not needed. Therefore, Datadog has set the limit at 350 metrics.

To see what you’re collecting and get below the limit, begin by using the commands seen above to investigate what metrics are available. It is recommended to create filters to refine what metrics are collected. If you believe you need more than 350 metrics, contact Datadog support.

Java Path

The Agent does not come with a bundled JVM, but uses the one installed on your system. Therefore you must make sure that the Java home directory is present in the path of the user running the Agent.

Alternatively, you can specify the JVM path in the integration’s configuration file:

java_bin_path: /path/to/java

Monitoring JBoss/WildFly applications

The following instructions work on Agent v5.6.0+.

JBoss/WildFly applications expose JMX over a specific protocol (Remoting JMX) that is not bundled by default with JMXFetch. To allow JMXFetch to connect to these applications, configure it as follows:

  • Locate the jboss-cli-client.jar file on your JBoss/WildFly server (by default, its path should be $JBOSS_HOME/bin/client/jboss-cli-client.jar).
  • If JMXFetch is running on a different host than the JBoss/WildFly application, copy jboss-cli-client.jar to a location on the host JMXFetch is running on.
  • Add the path of the jar to the init_config section of your configuration:
  # Datadog Agent >= 5.6.0

  init_config:
    custom_jar_paths:
      - /path/to/jboss-cli-client.jar
  • Specify a custom URL that JMXFetch connects to, in the instances section of your configuration:
  # Datadog Agent >= 5.6.0

  # The jmx_url may be different depending on the version of JBoss/WildFly you're using
  # and the way you've set up JMX on your server
  # Refer to the relevant documentation of JBoss/WildFly for more information
  instances:
    - jmx_url: "service:jmx:remote://localhost:4447"
      name: jboss-application  # Mandatory, but can be set to any value,
                               # is used to tag the metrics pulled from that instance

Monitoring Tomcat with JMX Remote Lifecycle Listener enabled

The following instructions work on Agent v5.6.0+.

If you’re using Tomcat with JMX Remote Lifecycle Listener enabled (see the Tomcat documentation for more information), JMXFetch needs additional setup to be able to connect to your Tomcat application.

  • Locate the catalina-jmx-remote.jar file on your Tomcat server (by default, its path should be $CATALINA_HOME/lib).
  • If JMXFetch is running on a different host than the Tomcat application, copy catalina-jmx-remote.jar to a location on the host JMXFetch is running on.
  • Add the path of the jar to the init_config section of your configuration:
init_config:
  custom_jar_paths:
    - /path/to/catalina-jmx-remote.jar
  • Specify a custom URL that JMXFetch connects to, in the instances section of your configuration:
# Datadog Agent >= 5.6.0

# The jmx_url may be different depending on the way you've set up JMX on your Tomcat server
instances:
  - jmx_url: "service:jmx:rmi://:10002/jndi/rmi://:10001/jmxrmi"
    name: tomcat-application  # Mandatory, but can be set to any arbitrary value,
                              # is used to tag the metrics pulled from that instance

Further Reading