--- title: Troubleshoot Missing Log Bytes description: >- Investigate unread log data reported after file rotation and identify backpressure in the Datadog Agent logs pipeline. breadcrumbs: Docs > Agent > Agent Troubleshooting > Troubleshoot Missing Log Bytes --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Troubleshoot Missing Log Bytes ## Overview{% #overview %} The Datadog Agent collects application logs and sends them to Datadog. For applications that write logs to files, the Agent reads those files as new entries are added. Applications often rotate their log files, replacing the active file with a new one when it reaches a size or age limit. If the Agent stops reading a rotated file before reaching the end, the remaining data is not sent to Datadog. The Agent reports this as **missing log bytes**. You may notice gaps in [Log Explorer](https://docs.datadoghq.com/logs/explorer.md) or see a warning in the [Agent log file](https://docs.datadoghq.com/agent/configuration/agent-log-files.md): ```text WARN | After rotation close timeout (60s), there were 148213 bytes remaining unread for file "/var/log/app/app.log". These unread logs are now lost. Consider increasing DD_LOGS_CONFIG_CLOSE_TIMEOUT ``` The Agent continues reading a rotated file for `logs_config.close_timeout` (60 seconds by default) after detecting rotation. If unread data remains when this timeout expires, the Agent closes the file and writes the warning above. ## Possible causes{% #possible-causes %} Backpressure occurs when a stage in the Agent's logs pipeline cannot keep up, slowing earlier stages and file reads. Frequent rotation or high log volume can also prevent the Agent from finishing within the timeout. - The Agent cannot deliver payloads to the Datadog intake because of network, proxy, authentication, or intake errors. - Network or intake latency slows successful log submissions. - Encoding, batching, or compression is limiting a logs pipeline. - Processing rules or multi-line processing are limiting a logs pipeline. - The log rate exceeds the CPU or network capacity available to the Agent. - The affected file rotates before the Agent can read the data written during the rotation interval, even when the downstream pipeline is not saturated. ## Diagnose missing log bytes{% #diagnose-missing-log-bytes %} 1. Search the [Agent log file](https://docs.datadoghq.com/agent/configuration/agent-log-files.md) for the rotation warning. Record the affected file paths, timestamps, and number of unread bytes. 1. If you use Agent 7.82.0 or later, run the [status command](https://docs.datadoghq.com/agent/configuration/agent-commands.md#agent-status-and-information) and read the Interpret Logs Agent Backpressure section. Compare the retained history with the warning timestamps. 1. If several rows in the **Logs Agent Backpressure** table are saturated, investigate them in this order: `destination_reliable_N`, `worker`, `strategy`, and `processor`. Use the Choose a tuning action section to select the next step for the first saturated component in that list. 1. Apply one relevant change, then follow the Verify the result section under representative log volume before addressing another component. 1. If the Agent remains unable to process the required volume, follow the Reduce log volume section. ### Interpret Logs Agent Backpressure{% #interpret-logs-agent-backpressure %} Agent versions 7.82.0 and later include a **Logs Agent Backpressure** section in the [status command](https://docs.datadoghq.com/agent/configuration/agent-commands.md#agent-status-and-information). The section reports recent utilization for the processor, batching strategy, sender workers, and destinations. It retains peak utilization for approximately 10 hours and detailed saturation duration for 30 minutes. ```shell sudo datadog-agent status ``` You can also find the section in `status.log` inside [an Agent flare](https://docs.datadoghq.com/agent/troubleshooting/send_a_flare.md). Because the history resets when the Agent restarts, take a flare before restarting. ```text Logs Agent Backpressure ======================= Overall state: SATURATED Reason: destination_reliable_0 pipeline q0s0 is currently saturated (saturated for 3m20s in the last 30m) Component Instance Current 5m avg/max 30m avg/max 2h max 5h max 10h max 30m saturated Last saturated processor 0 20% 18/26% 15/41% 55% 55% 55% 0s - strategy 0 14% 12/19% 13/22% 22% 22% 22% 0s - worker q0s0 46% 43/63% 27/63% 63% 63% 63% 0s - destination_reliable_0 q0s0 93% 82/93% 71/93% 93% 93% 93% 3m20s 12:09:42 ``` Because `destination_reliable_0` is saturated, follow the Resolve delivery errors section before changing processor or batching settings. #### Overall state{% #overall-state %} | State | Meaning | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `HEALTHY` | No component is saturated and none was saturated in the last 30 minutes. | | `WARNING` | No component is saturated, but at least one was saturated in the last 30 minutes. Compare `Last saturated` with the missing-bytes warning timestamps. | | `SATURATED` | At least one component reached the saturation threshold within the current 15-second window. | Saturation means that a component spent at least 90% of its sampled time working instead of waiting. Use the rotation warning to confirm loss and the table to identify a possible constraint. #### Table columns{% #table-columns %} | Column | Meaning | | --------------------------------- | ------------------------------------------------------------------------------------------------ | | `Component` | The logs pipeline stage. See the Choose a tuning action section. | | `Instance` | The pipeline or destination instance. | | `Current` | Utilization smoothed over approximately 15 seconds. | | `5m avg/max` and `30m avg/max` | Average and peak utilization during the stated window. | | `2h max`, `5h max`, and `10h max` | Peak utilization during the stated window. | | `30m saturated` | Time spent at or above 90% utilization during the last 30 minutes. | | `Last saturated` | Time of the latest retained saturation sample. A `-` means that no retained sample is available. | ## Give the Agent more time to read rotated files{% #give-the-agent-more-time-to-read-rotated-files %} Increase `logs_config.close_timeout` when the Agent needs more than the default 60 seconds to finish reading a rotated file. To use a 180-second timeout: {% tab title="Configuration file" %} In the `datadog.yaml` file: ```yaml logs_config: close_timeout: 180 ``` {% /tab %} {% tab title="Environment variable" %} ```shell DD_LOGS_CONFIG_CLOSE_TIMEOUT=180 ``` {% /tab %} After changing the timeout, search the [Agent log file](https://docs.datadoghq.com/agent/configuration/agent-log-files.md) for new rotation warnings during representative log volume. If warnings continue, increase the timeout again or use the Choose a tuning action section to select an action for the saturated component. Longer timeouts keep rotated files open and can increase file descriptor and disk usage. If the Agent does not tail all matching files, check `logs_config.open_files_limit` instead. For configuration details, see [Increase the Number of Log Files Tailed by the Agent](https://docs.datadoghq.com/logs/guide/increase-number-of-log-files-tailed.md). ## Tune the Agent{% #tune-the-agent %} ### Choose a tuning action{% #choose-a-tuning-action %} | Component | Interpretation | Start with this section | | ------------------------ | ------------------------------------------------------- | ------------------------------------------------------ | | `destination_reliable_N` | Log submissions are delayed or retried. | Resolve delivery errors | | `worker` | Payloads are waiting to be sent. | Resolve delivery errors | | `strategy` | Encoding, batching, or compression work is at capacity. | Disable compression or increase pipeline parallelism | | `processor` | Log processing is at capacity. | Reduce log processing or increase pipeline parallelism | ### Resolve delivery errors{% #resolve-delivery-errors %} If `destination_reliable_N` is saturated, open the [Agent log file](https://docs.datadoghq.com/agent/configuration/agent-log-files.md) and inspect entries near the missing-byte warning timestamp for failed or retried submissions. Resolve authentication, rejection, proxy, DNS, and connection errors. Verify the [proxy configuration](https://docs.datadoghq.com/agent/configuration/proxy.md), [network access to Datadog endpoints](https://docs.datadoghq.com/agent/configuration/network.md), and [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ### Disable compression{% #disable-compression %} For HTTPS delivery, compression is enabled by default. If `strategy` is saturated and the Agent's CPU usage is high, consider disabling compression. This reduces compression work, but increases payload size and the number of bytes sent to the logs intake: In the `datadog.yaml` file: ```yaml logs_config: use_compression: false ``` After the change, run the [status command](https://docs.datadoghq.com/agent/configuration/agent-commands.md#agent-status-and-information) and confirm that `strategy` spends less time saturated. Re-enable compression if network use reaches capacity or the [Agent log file](https://docs.datadoghq.com/agent/configuration/agent-log-files.md) shows new delivery errors. ### Increase pipeline parallelism{% #increase-pipeline-parallelism %} By default, the Agent runs one logs pipeline per available logical CPU, up to four pipelines. If `processor` or `strategy` is saturated and the host CPU is below capacity, increase `logs_config.pipelines`. For example, on a host with eight logical CPUs available to the Agent: In the `datadog.yaml` file: ```yaml logs_config: pipelines: 8 ``` Additional pipelines help when the Agent collects from multiple log sources. A single source remains on one pipeline. More pipelines can increase Agent CPU, memory, and total HTTP send concurrency. After the change, run the [status command](https://docs.datadoghq.com/agent/configuration/agent-commands.md#agent-status-and-information) and confirm that the targeted component spends less time saturated. ### Reduce log processing{% #reduce-log-processing %} If `processor` is saturated, review the configured [processing rules](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md). Global rules run against every log collected by the Agent. When a rule applies to one source, move it to that source's integration configuration: In the `conf.d/myapp.d/conf.yaml` file: ```yaml logs: - type: file path: /var/log/app/app.log service: myapp source: myapp log_processing_rules: - type: exclude_at_match name: exclude_debug pattern: \[DEBUG\] ``` Remove duplicate rules from global `logs_config.processing_rules` and source-level `log_processing_rules`. Starting with Agent 7.82.0, [automatic multi-line detection](https://docs.datadoghq.com/agent/logs/auto_multiline_detection.md) is enabled by default. If the `processor` component is saturated and your logs do not require multi-line aggregation, consider disabling it: In the `datadog.yaml` file: ```yaml logs_config: auto_multi_line_detection: false ``` If a source has a known multi-line format, [configure a source-level `multi_line` processing rule](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md#manually-aggregate-multi-line-logs) instead of using automatic detection. After the change, run the [status command](https://docs.datadoghq.com/agent/configuration/agent-commands.md#agent-status-and-information) to compare processor saturation and verify in [Log Explorer](https://docs.datadoghq.com/logs/explorer.md) that related lines remain grouped correctly. ## Reduce log volume{% #reduce-log-volume %} If the Agent cannot process the required volume within the available host and network capacity, filter logs that you do not need. An `exclude_at_match` processing rule drops matching logs before they leave the host: In the `datadog.yaml` file: ```yaml logs_config: processing_rules: - type: exclude_at_match name: exclude_health_checks pattern: GET /health ``` Use a source-level `log_processing_rules` entry instead when the filter applies to only one log source. ## Verify the result{% #verify-the-result %} Test each change under representative log volume: 1. Run the status command: ```shell sudo datadog-agent status ``` 1. Confirm that the targeted component spends less time at or above 90% utilization. The overall state remains `WARNING` for up to 30 minutes after saturation clears. 1. Search the [Agent log file](https://docs.datadoghq.com/agent/configuration/agent-log-files.md) for new `remaining unread` warnings. If a change does not reduce saturation or the frequency of new rotation warnings, revert it. Then select the next action from the Choose a tuning action section. If saturation moves to a different component, use the action listed for that component. ## Contact Datadog Support{% #contact-datadog-support %} If the steps on this page do not resolve the issue, [contact Datadog Support](https://docs.datadoghq.com/help/). Submit [a flare](https://docs.datadoghq.com/agent/troubleshooting/send_a_flare.md) while the pipeline is saturated and before restarting the Agent. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Agent Transport for Logs](https://docs.datadoghq.com/agent/logs/log_transport.md) - [Advanced Log Collection](https://docs.datadoghq.com/agent/logs/advanced_log_collection.md) - [Mechanisms to Ensure Logs are Not Lost](https://docs.datadoghq.com/logs/guide/mechanisms-ensure-logs-not-lost.md) - [Send an Agent Flare](https://docs.datadoghq.com/agent/troubleshooting/send_a_flare.md)