This product is not supported for your selected
Datadog site. (
).
Overview
Size your BYOC (Bring Your Own Cloud) Logs cluster in three steps:
- Estimate your daily ingest volume in TB/day.
- Pick a starter configuration for that volume.
- Monitor the cluster and adjust replica counts and pod sizes.
Searcher capacity depends on query concurrency, query complexity, and the amount of data scanned, not on ingest volume alone.
These recommendations assume modern x86 CPUs, such as those used in AWS M6 instance types, or equivalent CPUs from other cloud providers. ARM-based CPUs, such as AWS Graviton, may provide better cost efficiency at comparable throughput.
Starter configurations
Use these totals as a starting point:
- Indexers: 2 vCPUs per TB/day
- Compactors: 1 vCPU per 2 TB/day
- Searchers: about twice the indexer vCPU total. Analytics-heavy workloads may need up to twice the values shown in the table.
Object storage totals assume 30-day retention and a 6x compression ratio.
| Daily volume | Indexers | Compactors | Searchers | Object storage |
|---|
| 1 TB/day | 2 vCPUs | 0.5 vCPUs | 4 vCPUs | ~5 TB |
| 10 TB/day | 20 vCPUs | 5 vCPUs | 40 vCPUs | ~50 TB |
| 100 TB/day | 200 vCPUs | 50 vCPUs | 400 vCPUs | ~500 TB |
Recommended size for each pod:
| Daily volume | Indexers | Compactors | Searchers |
|---|
| Up to 30 TB/day | 4 vCPUs, 16 GB | 4 vCPUs, 16 GB | 16 vCPUs, 64 GB |
| Above 30 TB/day | 8 vCPUs, 32 GB | 8 vCPUs, 32 GB | 64 vCPUs, 256 GB |
Billing vs. provisioning: Provisioned vCPUs and billed vCPUs are different. A production cluster is intentionally overprovisioned to absorb ingestion and search spikes. Contact your Datadog representative for billing guidance.
Size each component
Adjust the starter configuration component by component. For the role each component plays, see Architecture.
Indexers
If you know your daily event count but not your byte volume, use this formula to estimate:
$$\text"Daily volume (TB)" = {\text"events per day" × \text"average event size (bytes)"} / 10^{12}$$
For example, with 1 billion events/day at 1 KB average size:
1,000,000,000 × 1,000 / 1,000,000,000,000 = 1 TB/day
Typical log event sizes range from 500 bytes (short syslog) to 2-3 KB (JSON with Kubernetes tags). Measure a representative sample of your logs to get an accurate average.
Compactors
- Performance: 1 vCPU per 2 TB/day
- Memory: 4 GB RAM per vCPU
- Storage type: Local SSD. Use instances with local SSDs, such as AWS M8gd.
Searchers
Size searchers for the expected search workload, not ingest volume alone. A starting point is about twice the indexer vCPU total.
- Performance: Term queries (
status:error AND message:exception) usually use less CPU than wildcard or whole-event searches. Aggregation queries need more CPU and memory. - Memory: 4 GB RAM per searcher vCPU. Provision more RAM if you expect many concurrent aggregation requests.
If search latency is high, add searcher replicas or increase memory per pod. See Scale searchers based on your query patterns.
Other services
Allocate the following resources for these lightweight components:
| Service | vCPUs | RAM | Replicas |
|---|
| Control Plane | 2 | 4 GB | 1 |
| Metastore | 2 | 4 GB | 2 |
| Janitor | 2 | 4 GB | 1 |
PostgreSQL database
- Instance size: For most use cases, a PostgreSQL instance with 1 vCPU and 4 GB of RAM is sufficient.
- Amazon RDS recommendation: On Amazon RDS, start with the
t4g.medium instance type. - High availability: Enable Multi-AZ deployment with one standby replica.
Enable automated backups on the metastore database. See Enable automated backups on your metastore database.
Object storage
BYOC Logs compresses and indexes log data before storing it in object storage. Compression is typically 5x to 8x, which translates to about 125-200 GB stored per TB ingested per day.
$$\text"Stored data per day" = {\text"Daily volume"} / {\text"compression ratio"}$$
$$\text"Total storage" = \text"Stored data per day" × \text"retention period (days)"$$
Use standard-tier object storage (for example, S3 Standard or GCS Standard) for active data. Lower-cost tiers such as S3 Infrequent Access or GCS Nearline are not validated for use with BYOC Logs.
To estimate PUT request volume and cost, see Object Storage Request Estimation.
Helm chart sizing tiers
Set indexer.podSize and searcher.podSize to match the per-pod CPU and memory in the starter configuration. The default is xlarge. Each preset also applies ingest queue and search cache sizes.
podSize | CPU | Memory |
|---|
large | 2 | 8Gi |
xlarge | 4 | 16Gi |
2xlarge | 8 | 32Gi |
4xlarge | 16 | 64Gi |
6xlarge | 24 | 96Gi |
8xlarge | 32 | 128Gi |
Each podSize requests less than its nominal CPU and memory, to leave room for kube-system, DaemonSets, and add-ons. The reservation amounts follow the GKE node reservation calculation, plus 250m CPU and 512Mi memory per node for DaemonSets and add-ons.
podSize | Actual CPU request | Actual memory request/limit |
|---|
large | 1600m | 5700Mi |
xlarge | 3600m | 13100Mi |
2xlarge | 7600m | 28500Mi |
4xlarge | 15600m | 59300Mi |
6xlarge | 23600m | 90100Mi |
8xlarge | 31600m | 120900Mi |
Actual CPU request = (nominal pod CPU - Kubernetes system CPU reservation - 250m), rounded down to the nearest 100m
Actual memory request/limit = (nominal pod memory - Kubernetes system memory reservation - 512Mi), rounded down to the nearest 100Mi
See the Helm chart sizing map for the complete configuration.
Further reading
Additional helpful documentation, links, and articles:
