JMX
Security Monitoring is now available Security Monitoring is now available

JMX

Agent Check Agent Check

Overview

The Java integration allows you to collects metrics, traces, and logs from your Java application.

Setup

Metric collection

If your application exposes JMX metrics, a lightweight Java plugin named JMXFetch (only compatible with Java >= 1.7.) is called by the Datadog Agent to connect to the MBean Server and collect your application metrics. It also sends service checks that report on the status of your monitored instances. This plugin sends metrics to the Datadog Agent using the DogStatsD server running within the Agent. These integrations also use the JMX metrics:

  • ActiveMQ
  • Cassandra
  • Solr
  • Tomcat
  • Kafka

Note: By default, JMX checks have a limit of 350 metrics per instance. If you require additional metrics, contact Datadog support.

Installation

Make sure you can open a JMX remote connection. A remote connection is required for the Datadog Agent to connect to the JVM, even when the two are on the same host.

Configuration

If running the Agent as a binary on a host, configure your JMX check as any other Agent integrations. If running the Agent as a DaemonSet in Kubernetes, configure your JMX check using auto-discovery.

Configure the Agent to connect using JMX and edit it according to your needs. Here is a sample jmx.d/conf.yaml file:

init_config:
  #custom_jar_paths:
  #  - <CUSTOM_JAR_FILE_PATH>.jar
  #is_jmx: true

instances:
  - host: "<JMX_INSTANCE_ENDPOINT>"
    port: "<JMX_PORT>"
    user: "<USER_NAME>"
    password: "<PASSWORD>"

    jmx_url: "service:jmx:rmi:///jndi/rmi://myhost.host:9999/<CUSTOM_PATH>" # optional

    name: "<JMX_INSTANCE_NAME>"
    java_bin_path: "<JAVA_PATH>"
    java_options: "-Xmx200m -Xms50m"
    trust_store_path: "<TRUST_STORE_PATH>.jks"
    trust_store_password: "<PASSWORD>"

    rmi_connection_timeout: 20
    rmi_client_timeout: 15000

    process_name_regex: ".*<PROCESS_NAME>.*"
    tools_jar_path: /usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar
    refresh_beans: 600
    tags:
      env: dev
      "<TAG_KEY>":"<TAG_VALUE>"

    conf:
      - include:
          domain: "<DOMAIN_NAME_1>"
          tags:
              simple: $attr0
              raw_value: "<CHOOSEN_VALUE>"
              multiple: $attr0-$attr1
          bean:
            - "<BEAN_NAME_1>"
            - "<BEAN_NAME_2>"
          attribute:
            attribute1:
              metric_type: counter
              alias: "jmx.<METRIC_NAME_ATTRIBUTE_1>"
            attribute2:
              metric_type: gauge
              alias: "jmx.<METRIC_NAME_ATTRIBUTE_2>"
            attribute3:
              metric_type: monotonic_count
              alias: "jmx.<METRIC_NAME_ATTRIBUTE_3>"

      - include:
          domain: "<DOMAIN_NAME_2>"
        exclude:
          bean:
            - "<EXCLUDED_BEAN_NAME>"
      - include:
          domain_regex: "<DOMAIN_REGEX>"
        exclude:
          bean_regex:
            - "<EXCLUDED_BEAN_NAME_REGEX>"
      - include:
          bean_regex: regex_topic=(.*?)
          attribute:
            attribute1:
              metric_type: gauge
              alias: "jmx.<ATTRIBUTE_NAME_WITH_REGEX_TAG>"

          ## The following submits jmx.<ATTRIBUTE_NAME_WITH_REGEX_TAG> bean with tags:
          ## `hostregex:<beanParameter>`
          ## `typeregex:<beanParameter>`
          ## `contextregex<beanParameter>`
          ## `optional:tag`
          tags:
              TypeRegex: $1
              HostRegex: $2
              contextRegex: $3
              optional: tag

Note: To run more than one JMX check, create configuration files with the format jmx_<INDEX>.d/conf.yaml (e.g. jmx_1.d/conf.yaml, jmx_2.d/conf.yaml, etc). Each folder should be stored in the conf.d directory. Include the is_jmx option set to true in those configuration files.

The standard datadog/agent:latest image for running the Datadog Agent container does not have JMX installed. Use the datadog/agent:latest-jmx image, this image is based on datadog/agent:latest, but it includes a JVM, which the Agent needs to run jmxfetch.

To run a JMX Check against one of your container:

  1. Create a JMX check configuration file by referring to the Host, or by using a JMX check configuration file for one of Datadog officially supported JMX integration:

  2. Mount this file inside the conf.d/ folder of your Datadog Agent: -v <HOST_FOLDER_PATH>:/conf.d. See the Setting up Check Templates documentation to learn more.

Note: Using %%port%% has proven problematic in practice. If you experience an issue, the best workaround is to replace %%port%% with a hard-coded JMX port.

Configuration Options
OptionRequiredDescription
custom_jar_pathsNoAllows specifying custom jars that are added to the classpath of the Agent’s JVM.
jmx_urlNoIf the Agent needs to connect to a non-default JMX URL, specify it here instead of a host and port. If you use this you need to specify a name for the instance.
is_jmxNoAllows creating different configuration files for each application rather than using a single long JMX file. Include the option in each configuration file as explained in the note from the Configuration section.
collect_default_metricsNoEach integration contains a metrics.yaml file that contains a list of default beans to collect. Setting this to True automatically collects those metrics without explicitly adding them to the yaml file. This is typically used for setting up the configuration with Autodiscovery to reduce the size of the configuration object.
nameNoUsed in conjunction with jmx_url.
java_bin_pathNoShould be set if the Agent cannot find your Java executable.
java_optionsNoJava JVM options
trust_store_path and trust_store_passwordNoShould be set if SSL is enabled.
process_name_regexNoInstead of specifying a host and port or jmx_url, the Agent can connect using the attach API. This requires the JDK to be installed and the path to tools.jar to be set.
tools_jar_pathNoTo be set when process_name_regex is set.
refresh_beansNoRefresh period for refreshing the matching MBeans list. Default is 600 seconds. Decreasing this value may result in increased CPU usage.
rmi_connection_timeoutNoThe connection timeout, in seconds, when connecting to a JVM using host and port or a jmx_url.
rmi_client_timeoutNoSpecify the duration without reply from the connected JVM, in milliseconds, after which the Agent will give up on an exiting connection and retry.

The conf parameter is a list of dictionaries. Only 2 keys are allowed in this dictionary:

KeyRequiredDescription
includeYesA dictionary of filters - any attribute that matches these filters are collected unless it also matches the “exclude” filters (see below).
excludeNoA dictionary of filters - attributes that match these filters are not collected.

Tags are automatically added to metrics based on the actual MBean name. You can explicitly specify supplementary tags. For instance, assuming the following MBean is exposed by your monitored application:

mydomain:attr0=val0,attr1=val1

It would create a metric called mydomain (or some variation depending on the attribute inside the bean) with tags: attr0:val0, attr1:val1, domain:mydomain, simple:val0, raw_value:my_chosen_value, multiple:val0-val1.

If you specify an alias in an include key that is formatted as camel case, it is converted to snake case. For example, MyMetricName is shown in Datadog as my_metric_name.

Description of the filters

Each include or exclude dictionary supports the following keys:

KeyDescription
domainA list of domain names (e.g. java.lang).
domain_regexA list of regexes on the domain name (e.g. java\.lang.*).
bean or bean_nameA list of full bean names (e.g. java.lang:type=Compilation).
bean_regexA list of regexes on the full bean names (e.g. java\.lang.*[,:]type=Compilation.*). You can use capture groups in your regex to supply as tag values. See example configuration above.
attributeA list or a dictionary of attribute names (see below for more details).

The regexes defined in domain_regex and bean_regex must conform to Java’s regular expression format.

The domain_regex and bean_regex filters were added in version 5.5.0.

On top of these parameters, the filters support “custom” keys which allows you to filter by bean parameters. For example, if you want to collect metrics regarding the Cassandra cache, you could use the type: - Caches filter:

conf:
    - include:
          domain: org.apache.cassandra.db
          type:
              - Caches

The Attribute filter

The attribute filter can accept two types of values:

  • A dictionary whose keys are attributes names:

    conf:
        - include:
              attribute:
                  maxThreads:
                      alias: tomcat.threads.max
                      metric_type: gauge
                  currentThreadCount:
                      alias: tomcat.threads.count
                      metric_type: gauge
                  bytesReceived:
                      alias: tomcat.bytes_rcvd
                      metric_type: counter

    In that case you can specify an alias for the metric that becomes the metric name in Datadog. You can also specify the metric type: gauge, histogram, counter/rate, or monotonic_count. If you choose counter, a rate per second is computed for the metric and it is sent as a gauge.

  • A list of attributes names:

    conf:
        - include:
              domain: org.apache.cassandra.db
              attribute:
                  - BloomFilterDiskSpaceUsed
                  - BloomFilterFalsePositives
                  - BloomFilterFalseRatio
                  - Capacity
                  - CompressionRatio
                  - CompletedTasks
                  - ExceptionCount
                  - Hits
                  - RecentHitRate

    In that case:

    • The metric type is a gauge.
    • The metric name is jmx.<DOMAIN_NAME>.<ATTRIBUTE_NAME>.

Here is another filtering example:

instances:
    - host: 127.0.0.1
      name: jmx_instance
      port: 9999

init_config:
    conf:
        - include:
              bean: org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency
              attribute:
                  - OneMinuteRate
                  - 75thPercentile
                  - 95thPercentile
                  - 99thPercentile

Validation

Run the Agent’s status subcommand and look for your JMX check under the JMXFetch section.

Additionally, JMX checks have a default configuration that collect 11 metrics from your JMX application. Check the Metrics Explorer for: jvm.heap_memory, jvm.non_heap_memory, or jvm.gc.cms.count.

Log collection

Available for Agent v6.0+

See the dedicated documentation on how to setup Java log collection to forward your logs to Datadog.

Trace collection

After enabling trace collection with your Agent, see the dedicated documentation for instrumenting your Java application to send its traces to Datadog.

Data Collected

Metrics

jvm.heap_memory
(gauge)
The total Java heap memory used.
Shown as byte
jvm.heap_memory_committed
(gauge)
The total Java heap memory committed to be used.
Shown as byte
jvm.heap_memory_init
(gauge)
The initial Java heap memory allocated.
Shown as byte
jvm.heap_memory_max
(gauge)
The maximum Java heap memory available.
Shown as byte
jvm.non_heap_memory
(gauge)
The total Java non-heap memory used.
Shown as byte
jvm.non_heap_memory_committed
(gauge)
The total Java non-heap memory committed to be used.
Shown as byte
jvm.non_heap_memory_init
(gauge)
The initial Java non-heap memory allocated.
Shown as byte
jvm.non_heap_memory_max
(gauge)
The maximum Java non-heap memory available.
Shown as byte
jvm.thread_count
(count)
The number of live threads.
Shown as thread
jvm.gc.cms.count
(count)
The total number of garbage collections that have occurred.
jvm.gc.major_collection_count
(count)
The number of major garbage collections that have occurred. Set `new_gc_metrics: true` to receive this metric.
jvm.gc.minor_collection_count
(count)
The number of minor garbage collections that have occurred. Set `new_gc_metrics: true` to receive this metric.
jvm.gc.parnew.time
(gauge)
The approximate accumulated garbage collection time elapsed.
Shown as millisecond
jvm.gc.major_collection_time
(gauge)
The approximate major garbage collection time elapsed. Set `new_gc_metrics: true` to receive this metric.
Shown as millisecond
jvm.gc.minor_collection_time
(gauge)
The approximate minor garbage collection time elapsed. Set `new_gc_metrics: true` to receive this metric.
Shown as millisecond

Note: Set new_gc_metrics: true in your jmx.d/conf.yaml to replace the following metrics:

jvm.gc.cms.count   => jvm.gc.minor_collection_count
                      jvm.gc.major_collection_count
jvm.gc.parnew.time => jvm.gc.minor_collection_time
                      jvm.gc.major_collection_time

Troubleshooting

Consult the list of JMX troubleshooting commands and FAQs.

Further Reading