--- title: Apache NiFi description: >- Monitor Apache NiFi data flow health, JVM performance, queue backpressure, and processor status. breadcrumbs: Docs > Integrations > Apache NiFi --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Apache NiFi Supported OS Integration version1.0.0 {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} ## Overview{% #overview %} This check monitors [Apache NiFi](https://nifi.apache.org/) through the Datadog Agent. Apache NiFi is a data integration and automation platform for moving data between systems. This integration uses the NiFi REST API to collect JVM health, flow throughput, queue backpressure, processor status, and bulletin events, and it collects application logs from `nifi-app.log`. Together, this data provides visibility into data pipeline health without requiring NiFi-side configuration. ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/containers/kubernetes/integrations.md) for guidance on applying these instructions. ### Installation{% #installation %} The NiFi check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package. No additional installation is needed on your server. ### Configuration{% #configuration %} 1. Edit the `nifi.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your NiFi performance data. See the [sample nifi.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/nifi/datadog_checks/nifi/data/conf.yaml.example) for all available configuration options. 1. At minimum, configure the `api_url`, `username`, and `password`: ```yaml instances: - api_url: https://localhost:8443/nifi-api username: password: tls_verify: true ``` 1. [Restart the Agent](https://docs.datadoghq.com/agent/configuration/agent-commands.md#start-stop-and-restart-the-agent). ### Log collection{% #log-collection %} 1. Collecting logs is disabled by default in the Datadog Agent. Enable it in your `datadog.yaml` file: ```yaml logs_enabled: true ``` 1. Add the following configuration block to your `nifi.d/conf.yaml` file to start collecting your NiFi logs: ```yaml logs: - type: file path: /opt/nifi/logs/nifi-app.log source: nifi service: nifi ``` Change the `path` and `service` parameter values based on your environment. See the [sample nifi.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/nifi/datadog_checks/nifi/data/conf.yaml.example) for all available configuration options. 1. Optional: Configure NiFi to include a UTC offset in its log timestamps. By default, NiFi writes local time without a timezone, so the integration keeps that time in the `nifi.local_time` attribute and dates each log by when Datadog received it. To date logs by the time NiFi wrote them, edit `conf/logback.xml` in your NiFi installation and set the pattern of the `APP_FILE` appender to: ```xml %date{"yyyy-MM-dd HH:mm:ss,SSSXXX"} %level [%thread] %logger{40} %msg%n ``` Keep the quotes around the date format. Without them, logback reads the text after the comma as a timezone ID and NiFi starts without any logging. Restart NiFi to apply the change. 1. [Restart the Agent](https://docs.datadoghq.com/agent/configuration/agent-commands.md#start-stop-and-restart-the-agent). ### Validation{% #validation %} [Run the Agent's status subcommand](https://docs.datadoghq.com/agent/configuration/agent-commands.md#agent-status-and-information) and look for `nifi` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **nifi.can_connect**(gauge) | Whether the check can reach and authenticate with the NiFi API. 1 = reachable 0 = unreachable. | | **nifi.cluster.connected_node_count**(gauge) | Number of nodes currently connected to the cluster.*Shown as node* | | **nifi.cluster.is_healthy**(gauge) | Whether all cluster nodes are connected. 1 = all connected 0 = degraded. | | **nifi.cluster.total_node_count**(gauge) | Total number of nodes in the cluster.*Shown as node* | | **nifi.connection.flowfiles_in**(gauge) | FlowFiles into this connection during snapshot window. | | **nifi.connection.flowfiles_out**(gauge) | FlowFiles out of this connection during snapshot window. | | **nifi.connection.percent_use_bytes**(gauge) | Backpressure utilization by size threshold.*Shown as percent* | | **nifi.connection.percent_use_count**(gauge) | Backpressure utilization by count threshold.*Shown as percent* | | **nifi.connection.queued_bytes**(gauge) | Bytes currently queued in this connection.*Shown as byte* | | **nifi.connection.queued_count**(gauge) | FlowFiles currently queued in this connection. | | **nifi.flow.active_threads**(gauge) | Active threads across all processors.*Shown as thread* | | **nifi.flow.bytes_queued**(gauge) | Total bytes queued across all connections.*Shown as byte* | | **nifi.flow.disabled_count**(gauge) | Number of disabled processors. | | **nifi.flow.flowfiles_queued**(gauge) | Total FlowFiles queued across all connections. | | **nifi.flow.invalid_count**(gauge) | Number of invalid processors. | | **nifi.flow.running_count**(gauge) | Number of running processors. | | **nifi.flow.stopped_count**(gauge) | Number of stopped processors. | | **nifi.process_group.active_threads**(gauge) | Active threads in this group.*Shown as thread* | | **nifi.process_group.bytes_queued**(gauge) | Bytes queued in this group.*Shown as byte* | | **nifi.process_group.bytes_read**(gauge) | Bytes read during snapshot window.*Shown as byte* | | **nifi.process_group.bytes_written**(gauge) | Bytes written during snapshot window.*Shown as byte* | | **nifi.process_group.flowfiles_queued**(gauge) | FlowFiles queued in this group. | | **nifi.process_group.flowfiles_received**(gauge) | FlowFiles received during snapshot window. | | **nifi.process_group.flowfiles_sent**(gauge) | FlowFiles sent during snapshot window. | | **nifi.process_group.flowfiles_transferred**(gauge) | FlowFiles transferred during snapshot window. | | **nifi.processor.active_threads**(gauge) | Active threads for this processor.*Shown as thread* | | **nifi.processor.bytes_read**(gauge) | Bytes read during snapshot window.*Shown as byte* | | **nifi.processor.bytes_written**(gauge) | Bytes written during snapshot window.*Shown as byte* | | **nifi.processor.flowfiles_in**(gauge) | FlowFiles received during snapshot window. | | **nifi.processor.flowfiles_out**(gauge) | FlowFiles emitted during snapshot window. | | **nifi.processor.processing_nanos**(gauge) | Processing time during snapshot window.*Shown as nanosecond* | | **nifi.processor.run_status**(gauge) | Processor run status. 1=Running 0=Stopped -1=Invalid -2=Disabled -3=Validating. | | **nifi.processor.task_count**(gauge) | Tasks executed during snapshot window. | | **nifi.system.content_repo.free_space**(gauge) | Content repository free space.*Shown as byte* | | **nifi.system.content_repo.used_space**(gauge) | Content repository used space.*Shown as byte* | | **nifi.system.content_repo.utilization**(gauge) | Content repository utilization.*Shown as percent* | | **nifi.system.cpu.available_processors**(gauge) | Number of available processors. | | **nifi.system.cpu.load_average**(gauge) | System processor load average. | | **nifi.system.flowfile_repo.free_space**(gauge) | FlowFile repository free space.*Shown as byte* | | **nifi.system.flowfile_repo.used_space**(gauge) | FlowFile repository used space.*Shown as byte* | | **nifi.system.flowfile_repo.utilization**(gauge) | FlowFile repository utilization.*Shown as percent* | | **nifi.system.gc.collection_count**(count) | GC collection count. | | **nifi.system.gc.collection_time**(count) | GC collection time.*Shown as millisecond* | | **nifi.system.jvm.daemon_threads**(gauge) | Daemon thread count.*Shown as thread* | | **nifi.system.jvm.heap_max**(gauge) | Maximum JVM heap memory.*Shown as byte* | | **nifi.system.jvm.heap_used**(gauge) | Current JVM heap memory used.*Shown as byte* | | **nifi.system.jvm.heap_utilization**(gauge) | JVM heap usage percentage.*Shown as percent* | | **nifi.system.jvm.non_heap_used**(gauge) | Current JVM non-heap memory used.*Shown as byte* | | **nifi.system.jvm.total_threads**(gauge) | Total JVM thread count.*Shown as thread* | | **nifi.system.provenance_repo.free_space**(gauge) | Provenance repository free space.*Shown as byte* | | **nifi.system.provenance_repo.used_space**(gauge) | Provenance repository used space.*Shown as byte* | | **nifi.system.provenance_repo.utilization**(gauge) | Provenance repository utilization.*Shown as percent* | Per-connection and per-processor metrics are opt-in to control cardinality. Enable them with `collect_connection_metrics: true` and `collect_processor_metrics: true`. Use `max_connections` and `max_processors` to cap the number of entities monitored. ### Events{% #events %} NiFi bulletins (errors and warnings from processors and system components) are forwarded as Datadog events when `collect_bulletins` is enabled (default: true). Filter by severity with `bulletin_min_level` (default: WARNING). ### Logs{% #logs %} The NiFi integration can collect application logs from `nifi-app.log`, including startup and shutdown events, processor errors, and audit trail entries. See Log collection for setup instructions. ### Service Checks{% #service-checks %} The NiFi integration does not include any service checks. Connectivity is reported via the `nifi.can_connect` gauge (1 = OK, 0 = unreachable). ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).