Processes

Supported OS Linux Mac OS Windows

Integration version3.3.0

Overview

The Process Check lets you:

  • Collect resource usage metrics for specific running processes on any host. For example, CPU, memory, I/O, and number of threads.
  • Use Process Monitors to configure thresholds for how many instances of a specific process should be running and get alerts when the thresholds aren’t met (see Service Checks below).

Setup

Installation

The Process check is included in the Datadog Agent package, so you don’t need to install anything else on your server.

Configuration

Unlike many checks, the Process check doesn’t monitor anything useful by default. You must configure which processes you want to monitor.

While there’s no standard default check configuration, here’s an example process.d/conf.yaml that monitors SSH/SSHD processes. See the sample process.d/conf.yaml for all available configuration options:

init_config:
instances:
- name: ssh
  search_string:
    - ssh
    - sshd

Note: After you make configuration changes, make sure you restart the Agent.

Retrieving some process metrics requires the Datadog collector to either run as the monitored process user or with privileged access. For the open_file_descriptors metric on Unix platforms, there is an additional configuration option. Setting try_sudo to true in your conf.yaml file allows the Process check to try using sudo to collect the open_file_descriptors metric. Using this configuration option requires setting the appropriate sudoers rules in /etc/sudoers:

dd-agent ALL=NOPASSWD: /bin/ls /proc/*/fd/

Validation

Run the Agent’s status subcommand and look for process under the Checks section.

Metrics notes

The following metrics are not available on Linux or macOS:

  • Process I/O metrics are not available on Linux or macOS since the files that the Agent reads (/proc//io) are only readable by the process’s owner. For more information, read the Agent FAQ.

The following metrics are not available on Windows:

  • system.cpu.iowait
  • system.processes.mem.page_faults.minor_faults
  • system.processes.mem.page_faults.children_minor_faults
  • system.processes.mem.page_faults.major_faults
  • system.processes.mem.page_faults.children_major_faults

Note: Use a WMI check to gather page fault metrics on Windows.

Note: In v6.11+ on Windows, the Agent runs as ddagentuser instead of Local System. Because of this, it does not have access to the full command line of processes running under other users and to the user of other users’ processes. This causes the following options of the check to not work:

  • exact_match when set to false
  • user, which allows selecting processes that belong to a specific user

All metrics are per instance configured in process.yaml, and are tagged process_name:<instance_name>.

The system.processes.cpu.pct metric sent by this check is only accurate for processes that live for more than 30 seconds. Do not expect its value to be accurate for shorter-lived processes.

For the full list of metrics, see the Metrics section.

Data Collected

Metrics

system.processes.cpu.pct
(gauge)
The CPU utilization of a process.
Shown as percent
system.processes.cpu.normalized_pct
(gauge)
The normalized CPU utilization of a process.
Shown as percent
system.processes.involuntary_ctx_switches
(gauge)
The number of involuntary context switches performed by this process.
Shown as event
system.processes.ioread_bytes
(gauge)
The number of bytes read from disk by this process. In Windows: the number of bytes read.
Shown as byte
system.processes.ioread_bytes_count
(count)
The number of bytes read from disk by this process. In Windows: the number of bytes read.
Shown as byte
system.processes.ioread_count
(gauge)
The number of disk reads by this process. In Windows: the number of reads by this process.
Shown as read
system.processes.iowrite_bytes
(gauge)
The number of bytes written to disk by this process. In Windows: the number of bytes written by this process.
Shown as byte
system.processes.iowrite_bytes_count
(count)
The number of bytes written to disk by this process. In Windows: the number of bytes written by this process.
Shown as byte
system.processes.iowrite_count
(gauge)
The number of disk writes by this process. In Windows: the number of writes by this process.
Shown as write
system.processes.mem.page_faults.minor_faults
(gauge)
In Unix/Linux and macOS: The number of minor page faults per second for this process.
Shown as occurrence
system.processes.mem.page_faults.children_minor_faults
(gauge)
In Unix/Linux and macOS: The number of minor page faults per second for children of this process.
Shown as occurrence
system.processes.mem.page_faults.major_faults
(gauge)
In Unix/Linux and macOS: The number of major page faults per second for this process.
Shown as occurrence
system.processes.mem.page_faults.children_major_faults
(gauge)
In Unix/Linux and macOS: The number of major page faults per second for children of this process.
Shown as occurrence
system.processes.mem.pct
(gauge)
The process memory consumption.
Shown as percent
system.processes.mem.real
(gauge)
The non-swapped physical memory a process has used and cannot be shared with another process (Linux only).
Shown as byte
system.processes.mem.rss
(gauge)
The non-swapped physical memory a process has used. aka "Resident Set Size".
Shown as byte
system.processes.mem.vms
(gauge)
The total amount of virtual memory used by the process. aka "Virtual Memory Size".
Shown as byte
system.processes.number
(gauge)
The number of processes.
Shown as process
system.processes.open_file_descriptors
(gauge)
The number of file descriptors used by this process (only available for processes run as the dd-agent user)
system.processes.open_handles
(gauge)
The number of handles used by this process.
system.processes.threads
(gauge)
The number of threads used by this process.
Shown as thread
system.processes.voluntary_ctx_switches
(gauge)
The number of voluntary context switches performed by this process.
Shown as event
system.processes.run_time.avg
(gauge)
The average running time of all instances of this process
Shown as second
system.processes.run_time.max
(gauge)
The longest running time of all instances of this process
Shown as second
system.processes.run_time.min
(gauge)
The shortest running time of all instances of this process
Shown as second

Events

The Process Check does not include any events.

Service Checks

process.up
Returns OK if the check is within the warning thresholds, CRITICAL if it’s outside of the critical thresholds, and WARNING if it’s outside of the warning thresholds.
Statuses: ok, warning, critical

Troubleshooting

Need help? Contact Datadog support.

Further Reading

To get a better idea of how (or why) to monitor process resource consumption with Datadog, check out this series of blog posts about it.