Overview

Object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage) bills both stored data and API requests. This page estimates the PUT requests generated on the write path by indexers and compactors.

Indexers

Each indexer runs several indexing pipelines, one per vCPU. Every 30 seconds (the default commit timeout), each pipeline uploads one index file (a split) to object storage:

$$\text"splits per pipeline per day" = {86400} / 30 = 2880$$

A 4-vCPU indexer runs 4 pipelines, so it uploads 11,520 splits per day.

Splits store compressed data (3x or more). At typical volumes, they stay below the 128 MiB multipart threshold and upload as a single PUT request. Larger splits use multipart upload and cost 3 or more PUT requests.

Compactors

Compactors merge 10 splits into 1, repeatedly, over three generations (gen 1, 2, 3). Merged splits are large (over 1 GB) and use multipart upload, so each one costs 3 or more PUT requests.

The 10:1 fan-in means every generation produces 10x fewer splits than the one before it. In total, compaction adds only about one-third more PUT requests on top of the indexer splits.

Request estimates

The following estimates assume:

• 8 MB/s per indexer vCPU, so a 4-vCPU indexer sustains 32 MB/s, or ~2.8 TB/day. See Cluster Sizing.
• 4 indexing pipelines per indexer, with the default 30-second commit timeout
• A PUT price of $0.005 per 1,000 requests (S3 Standard, us-east-1)

Each indexer uploads 11,520 splits per day. Compaction adds about one-third, for a total of ~15,360 PUT requests per indexer per day:

Further reading

Additional helpful documentation, links, and articles:

BYOC Logs Cluster SizingDOCUMENTATION Learn more about BYOC Logs ArchitectureDOCUMENTATION