Solr

Solr

Agent Check Agent Check

Linux Mac OS Windows OS Supported

Solr Graph

Overview

The Solr check tracks the state and performance of a Solr cluster. It collects metrics for the number of documents indexed, cache hits and evictions, average request times, average requests per second, and more.

Setup

Installation

The Solr check is included in the Datadog Agent package, so you don’t need to install anything else on your Solr nodes.

This check is JMX-based, so you need to enable JMX Remote on your Solr servers. Read the JMX Check documentation for more details.

Configuration

Host

To configure this check for an Agent running on a host:

  1. Edit the solr.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory. See the sample solr.d/conf.yaml for all available configuration options.

    init_config:
    
      ## @param is_jmx - boolean - required
      ## Whether or not this file is a configuration for a JMX integration.
      #
      is_jmx: true
    
      ## @param collect_default_metrics - boolean - required
      ## Whether or not the check should collect all default metrics.
      #
      collect_default_metrics: true
        
    instances:
      ## @param host - string - required
      ## Solr host to connect to.
      - host: localhost
    
        ## @param port - integer - required
        ## Solr port to connect to.
        port: 9999
    
  2. Restart the Agent.

List of metrics

The conf parameter is a list of metrics to be collected by the integration. Only 2 keys are allowed:

  • include (mandatory): A dictionary of filters, any attribute that matches these filters are collected unless it also matches the exclude filters (see below).
  • exclude (optional): A dictionary of filters, attributes that match these filters are not collected.

For a given bean, metrics get tagged in the following manner:

mydomain:attr0=val0,attr1=val1

In this example, your metric is mydomain (or some variation depending on the attribute inside the bean) and has the tags attr0:val0, attr1:val1, and domain:mydomain.

If you specify an alias in an include key that is formatted as camel case, it is converted to snake case. For example, MyMetricName is shown in Datadog as my_metric_name.

The attribute filter

The attribute filter can accept two types of values:

  • A dictionary whose keys are attributes names (see below). For this case, you can specify an alias for the metric that becomes the metric name in Datadog. You can also specify the metric type as a gauge or counter. If you choose counter, a rate per second is computed for the metric.

    conf:
      - include:
        attribute:
          maxThreads:
            alias: tomcat.threads.max
            metric_type: gauge
          currentThreadCount:
            alias: tomcat.threads.count
            metric_type: gauge
          bytesReceived:
            alias: tomcat.bytes_rcvd
            metric_type: counter
    
  • A list of attributes names (see below). For this case, the metric type is a gauge, and the metric name is jmx.\[DOMAIN_NAME].\[ATTRIBUTE_NAME].

    conf:
      - include:
        domain: org.apache.cassandra.db
        attribute:
          - BloomFilterDiskSpaceUsed
          - BloomFilterFalsePositives
          - BloomFilterFalseRatio
          - Capacity
          - CompressionRatio
          - CompletedTasks
          - ExceptionCount
          - Hits
          - RecentHitRate
    

Older versions

List of filters is only supported in Datadog Agent > 5.3.0. If you are using an older version, use singletons and multiple include statements instead.

# Datadog Agent > 5.3.0
  conf:
    - include:
      domain: domain_name
      bean:
        - first_bean_name
        - second_bean_name
# Older Datadog Agent versions
  conf:
    - include:
      domain: domain_name
      bean: first_bean_name
    - include:
      domain: domain_name
      bean: second_bean_name

Containerized

For containerized environments, see the Autodiscovery with JMX guide.

Log collection
  1. Collecting logs is disabled by default in the Datadog Agent, enable it in your datadog.yaml file:

     logs_enabled: true
    
  2. Solr uses the log4j logger by default. To customize the logging format, edit the server/resources/log4j2.xml file. By default, Datadog’s integration pipeline supports the following conversion pattern:

    %maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n
    

    Clone and edit the integration pipeline if you have a different format.

  3. Uncomment and edit the logs configuration block in your solr.d/conf.yaml file. Change the type, path, and service parameter values based on your environment. See the sample solr.d/solr.yaml for all available configuration options.

     logs:
       - type: file
         path: /var/solr/logs/solr.log
         source: solr
         # To handle multi line that starts with yyyy-mm-dd use the following pattern
         # log_processing_rules:
         #   - type: multi_line
         #     pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])
         #     name: new_log_start_with_date
    
  4. Restart the Agent.

See Datadog’s documentation for additional information on how to configure the Agent for log collection in Docker environments.

Validation

Run the Agent’s status subcommand and look for solr under the Checks section.

Data Collected

Metrics

solr.cache.evictions
(gauge)
The total number of cache evictions per second.
Shown as eviction
solr.cache.hits
(gauge)
The number of cache hits per second.
Shown as hit
solr.cache.inserts
(gauge)
The total number of cache inserts per second.
Shown as set
solr.cache.lookups
(gauge)
The total number of cache lookups per second.
Shown as get
solr.document_cache.evictions
(gauge)
The total number of cache evictions per second.
Shown as eviction
solr.document_cache.hits
(gauge)
The number of cache hits per second.
Shown as hit
solr.document_cache.inserts
(gauge)
The total number of cache inserts per second.
Shown as set
solr.document_cache.lookups
(gauge)
The total number of cache lookups per second.
Shown as get
solr.filter_cache.evictions
(gauge)
The total number of cache evictions per second.
Shown as eviction
solr.filter_cache.hits
(gauge)
The number of cache hits per second.
Shown as hit
solr.filter_cache.inserts
(gauge)
The total number of cache inserts per second.
Shown as set
solr.filter_cache.lookups
(gauge)
The total number of cache lookups per second.
Shown as get
solr.query_result_cache.evictions
(gauge)
The total number of cache evictions per second.
Shown as eviction
solr.query_result_cache.hits
(gauge)
The number of cache hits per second.
Shown as hit
solr.query_result_cache.inserts
(gauge)
The total number of cache inserts per second.
Shown as set
solr.query_result_cache.lookups
(gauge)
The total number of cache lookups per second.
Shown as get
solr.search_handler.request_times.50percentile
(gauge)
Request processing time in milliseconds (50percentile).
Shown as millisecond
solr.search_handler.request_times.75percentile
(gauge)
Request processing time in milliseconds (75percentile).
Shown as millisecond
solr.search_handler.request_times.95percentile
(gauge)
Request processing time in milliseconds (95percentile).
Shown as millisecond
solr.search_handler.request_times.98percentile
(gauge)
Request processing time in milliseconds (98percentile).
Shown as millisecond
solr.search_handler.request_times.99percentile
(gauge)
Request processing time in milliseconds (99percentile).
Shown as millisecond
solr.search_handler.request_times.999percentile
(gauge)
Request processing time in milliseconds (999percentile).
Shown as millisecond
solr.search_handler.request_times.one_minute_rate
(gauge)
Requests per second received over the past minutes.
Shown as request
solr.search_handler.request_times.mean_rate
(gauge)
Average number of requests received per second since the Solr core was first created.
Shown as request
solr.search_handler.request_times.mean
(gauge)
The average time per request.
Shown as millisecond
solr.search_handler.avg_requests_per_sec
(gauge)
Average number of requests received per second since the Solr core was first created.
Shown as request
solr.search_handler.avg_time_per_req
(gauge)
The average time per request.
Shown as millisecond
solr.search_handler.errors
(gauge)
Number of errors per second encountered by the handler.
Shown as error
solr.search_handler.requests
(gauge)
Number of requests per second processed by the handler.
Shown as request
solr.search_handler.time
(gauge)
The sum of all request processing times (in milliseconds) per second.
solr.search_handler.timeouts
(gauge)
Number of responses per second received with partial results.
Shown as timeout
solr.searcher.maxdoc
(gauge)
One greater than the largest possible document number (Solr version <= 6).
Shown as document
solr.searcher.maxdocs
(gauge)
One greater than the largest possible document number (Solr version >= 7).
Shown as document
solr.searcher.numdocs
(gauge)
The total number of indexed documents.
Shown as document
solr.searcher.warmup
(gauge)
The time spent warming up.
Shown as millisecond

Events

The Solr check does not include any events.

Service Checks

solr.can_connect
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored SolR instance. Returns OK otherwise.
Statuses: ok, critical

Troubleshooting

Commands to view the available metrics

The datadog-agent jmx command was added in version 4.1.0.

  • List attributes that match at least one of your instances configuration: sudo datadog-agent jmx list matching
  • List attributes that do match one of your instances configuration but that are not being collected because it would exceed the number of metrics that can be collected: sudo datadog-agent jmx list limited
  • List attributes expected to be collected by your current instances configuration: sudo datadog-agent jmx list collected
  • List attributes that don’t match any of your instances configuration: sudo datadog-agent jmx list not-matching
  • List every attributes available that has a type supported by JMXFetch: sudo datadog-agent jmx list everything
  • Start the collection of metrics based on your current configuration and display them in the console: sudo datadog-agent jmx collect

Further Reading

Parsing a string value into a number

If your jmxfetch returns only string values like false and true and you want to transform it into a Datadog gauge metric for advanced usages. For instance if you want the following equivalence for your jmxfetch:

"myJmxfetch:false" = myJmxfetch:0
"myJmxfetch:true" = myJmxfetch:1

You may use the attribute filter as follow:

# ...
attribute:
  myJmxfetch:
    alias: your_metric_name
    metric_type: gauge
    values:
      "false": 0
      "true": 1