Configure disk buffers for durable BYOC Logs ingestion

Docs > BYOC Logs > Operate BYOC Logs > Configure disk buffers for durable BYOC Logs ingestion

Diagram of durable BYOC Logs ingestion using Observability Pipelines disk buffers and the BYOC Logs engine.

Overview

BYOC Logs combines Observability Pipelines with the BYOC Logs engine. Observability Pipelines Workers can store logs in persistent disk buffers when the BYOC Logs engine is unavailable or cannot keep up with the incoming rate. This gives the ingestion path time to recover without immediately dropping logs.

A durable setup has four parts:

Enough buffer capacity for the backlog you want to retain
Disk buffering on the BYOC Logs destination
A persistent volume for each Worker’s buffer
Enough time for a terminating Worker to drain its buffer

Before you begin

This guide assumes that you have a BYOC Logs deployment with:

Observability Pipelines configured with a BYOC Logs destination
A BYOC Logs engine deployment

Size the disk buffer

A useful starting point is to decide how long Workers need to retain logs if the BYOC Logs engine is unavailable. The expected backlog can then be divided across the minimum number of Workers that remain active.

For example, consider a total ingress of 50 TB/day, 25 Workers, and a one-hour BYOC Logs engine outage:

Total backlog: 50 TB × 1 hour ÷ 24 hours = 2.08 TB
Buffer per Worker: 2.08 TB ÷ 25 Workers ≈ 83 GB

Rounding up the result leaves capacity for a longer incident or a higher ingress rate. With a 100 GB buffer on each Worker, the 25 Workers provide 2.5 TB of total buffer capacity. This retains approximately 1.2 hours of logs at 50 TB/day.

Disk buffer configuration

The following buffering options are available when editing the BYOC Logs destination in the Observability Pipelines UI or through the API:

Buffer type: Disk
Buffer size: The per-Worker capacity from your calculation
Behavior on full buffer: Block

For the 25-Worker example, the destination configuration looks like this:

"buffer": {
  "type": "disk",
  "max_size": 100000000000,
  "when_full": "block"
}

The max_size value is expressed in bytes (100 GB = 100,000,000,000 bytes). When the buffer is full, block keeps the Worker from discarding logs and allows backpressure to propagate to upstream applications.

Persistent storage for the buffer

Disk buffering is durable only when the storage outlives the Worker pod. In the 100 GB buffer example, the following Helm values assign a 120 GiB (about 129 GB) persistent volume to each Worker. The extra capacity leaves headroom above the buffer size:

persistence:
  enabled: true
  storageClassName: "<STORAGE_CLASS>"
  accessModes:
    - ReadWriteOnce
  size: 120Gi
  retentionPolicy:
    whenScaled: Retain
    whenDeleted: Retain

The persistent volume keeps buffered logs if a Worker pod restarts. The Retain policies also keep the volume when its Worker is scaled down or the StatefulSet is deleted.

Draining Workers before shutdown

During a scale-down or rollout, a Worker stops accepting new logs and sends its buffered logs to the BYOC Logs engine. If the Worker exits before the buffer is empty, the logs remain on its persistent volume until a replacement Worker reattaches that volume.

The terminationGracePeriodSeconds setting controls how long Kubernetes waits before terminating the pod. This gives the Worker time to empty its buffer before exiting.

This limit determines how long the buffer has to drain. You can estimate the required time from the drain throughput measured in your environment:

$$\text"drain time" = {\text"buffer size (MB)"} / {\text"drain throughput (MB/s)"}$$

$$\text"termination grace period" = \text"drain time" × \text"safety factor"$$

For a Worker draining a 100 GB buffer at 120 MB/s, a safety factor of 1.2 gives: