--- title: Cluster Sizing description: Learn about cluster sizing for BYOC Logs breadcrumbs: Docs > BYOC Logs > Operate BYOC Logs > Cluster Sizing --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Cluster Sizing {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} Proper cluster sizing helps ensure optimal performance, cost efficiency, and reliability for your BYOC (Bring Your Own Cloud) Logs deployment. Your sizing requirements depend on several factors including log ingestion volume, query patterns, retention period, and the complexity of your log data. The sizing examples below provide starting-point configurations for common daily log volumes. For deeper guidance on each component, see the sections that follow. Use your expected daily log volume and peak ingestion rates as starting points, then monitor your cluster's performance and adjust sizing as needed. ## Sizing examples{% #sizing-examples %} The following table provides starting-point configurations for common daily log volumes. These are baseline recommendations—adjust based on your observed performance. As a rule of thumb for a mixed workload, plan for around 12 vCPUs per TB/day ingested—4 vCPUs for indexers and 8 vCPUs for searchers. Heavy analytics workloads need 2x more. These vCPU recommendations assume modern x86 CPUs such as AWS m6 instance types (or equivalent on other clouds). ARM-based CPUs such as AWS Graviton can offer better cost efficiency at the same throughput. | Daily volume | Indexer pods | Indexer podSize | Searcher pods | Searcher podSize | Object storage (30-day retention, ~6x compression) | | -------------- | ------------ | --------------- | ------------- | ---------------- | -------------------------------------------------- | | **1 TB/day** | 2 | large | 2 | xlarge | ~5 TB | | **5 TB/day** | 5 | xlarge | 5 | 2xlarge | ~25 TB | | **10 TB/day** | 10 | xlarge | 5 | 4xlarge | ~50 TB | | **50 TB/day** | 25 | 2xlarge | 13 | 8xlarge | ~250 TB | | **100 TB/day** | 50 | 2xlarge | 25 | 8xlarge | ~500 TB | {% alert level="info" %} **Billing vs. provisioning:** Provisioned vCPUs and billed vCPUs are different. A production cluster is intentionally overprovisioned to absorb ingestion and search spikes. Contact your Datadog representative for billing guidance. {% /alert %} ## Indexers{% #indexers %} Indexers receive logs from Datadog Agents, then process, index, and store them as index files (called *splits*) in object storage. Proper sizing is critical for maintaining ingestion throughput and ensuring your cluster can handle your log volume. | Specification | Recommendation | Notes | | -------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Performance** | 5 MB/s per vCPU | Baseline throughput to determine initial sizing. Actual performance depends on log characteristics (size, number of attributes, nesting level) | | **Memory** | 4 GB RAM per vCPU | | **Minimum Pod Size** | 2 vCPUs, 8 GB RAM | Recommended minimum for indexer pods | | **Storage Capacity** | At least 250 GB | Required for temporary data while creating and merging index files | | **Storage Type** | Network-attached block storage | For example: Amazon EBS gp3, Azure Managed Disks, or GCP Persistent Disk. Data is temporarily stored in a write-ahead log (WAL) before being uploaded to object storage. The WAL is not replicated, so using local (ephemeral) SSDs increases the risk of losing a few minutes of data if the disk fails. Network-attached block storage provides built-in redundancy. | | **Disk I/O** | ~20 MB/s per vCPU | Equivalent to 320 IOPS per vCPU for Amazon EBS (assuming 64 KB IOPS) | {% collapsible-section %} #### Example: Sizing for 1 TB of logs per day To index 1 TB of logs per day (~11.6 MB/s), follow these steps: 1. **Calculate vCPUs:** `11.6 MB/s ÷ 5 MB/s per vCPU ≈ 2.3 vCPUs` 1. **Calculate RAM:** `2.3 vCPUs × 4 GB RAM ≈ 9 GB RAM` 1. **Add headroom:** Start with one indexer pod configured with **3 vCPUs, 12 GB RAM, and a 200 GB disk**. Adjust these values based on observed performance and redundancy needs. {% /collapsible-section %} {% collapsible-section %} #### Sizing by event count If you know your daily event count but not your byte volume, use this formula to estimate: $$\text"Daily volume (TB)" = {\text"events per day" × \text"average event size (bytes)"} / 10^{12}$$ For example, with 1 billion events/day at 1 KB average size: `1,000,000,000 × 1,000 / 1,000,000,000,000 = 1 TB/day` Typical log event sizes range from 500 bytes (short syslog) to 2-3 KB (JSON with Kubernetes tags). Measure a representative sample of your logs to get an accurate average. {% /collapsible-section %} ## Searchers{% #searchers %} Searchers handle search queries from the Datadog UI, reading metadata from the Metastore and fetching data from object storage. A general starting point is to provision roughly double the total number of vCPUs allocated to Indexers. See our sizing examples. - **Performance:** Search performance depends heavily on the workload (query complexity, concurrency, amount of data scanned). For instance, term queries (`status:error AND message:exception`) are usually computationally less expensive than wildcard or whole event search queries. - **Memory:** 4 GB of RAM per searcher vCPU. Provision more RAM if you expect many concurrent aggregation requests. ## Other services{% #other-services %} Allocate the following resources for these lightweight components: | Service | vCPUs | RAM | Replicas | | ----------------- | ----- | ---- | -------- | | **Control Plane** | 2 | 4 GB | 1 | | **Metastore** | 2 | 4 GB | 2 | | **Janitor** | 2 | 4 GB | 1 | ## Object storage estimation{% #object-storage-estimation %} BYOC Logs compresses and indexes log data before storing it in object storage. The compression ratio depends on the log format, structure, and redundancy in your data. | Metric | Typical range | | ------------------------------- | ----------------------------------- | | **Compression ratio** | 5x to 8x (raw input to stored size) | | **Storage per TB/day ingested** | 125-200 GB/day on object storage | To estimate your object storage requirements: $$\text"Stored data per day" = {\text"Daily volume"} / {\text"compression ratio"}$$ $$\text"Total storage" = \text"Stored data per day" × \text"retention period (days)"$$ {% collapsible-section %} #### Example: Storage for 10 TB/day with 30-day retention Assuming a 6x compression ratio: 1. **Stored per day:** `10 TB / 6 ≈ 1.67 TB/day` 1. **Total for 30 days:** `1.67 TB × 30 ≈ 50 TB` Use standard-tier object storage (for example, S3 Standard, GCS Standard) for active data. Lower-cost tiers such as S3 Infrequent Access or GCS Nearline are not validated for use with BYOC Logs. {% /collapsible-section %} ## PostgreSQL database{% #postgresql-database %} - **Instance Size:** For most use cases, a PostgreSQL instance with 1 vCPU and 4 GB of RAM is sufficient - **AWS RDS Recommendation:** If using AWS RDS, the `t4g.medium` instance type is a suitable starting point - **High Availability:** Enable Multi-AZ deployment with one standby replica for high availability ## Helm chart sizing tiers{% #helm-chart-sizing-tiers %} The BYOC Logs Helm chart provides predefined resource tiers through the `indexer.podSize` and `searcher.podSize` parameters. `podSize` selects the pod's resource requirements and related Quickwit tuning parameters. The default `podSize` is `xlarge` for both components. Each preset is designed to leave room on a matching node for Kubernetes system components, DaemonSets, and add-ons. The presets account for resources reserved for Kubernetes system components. The reservation amounts are based on the [GKE node reservation calculation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/plan-node-sizes#resource_reservations). An additional 250m CPU and 512Mi memory per node is reserved for DaemonSets and add-ons: ```text Actual CPU request = (nominal pod CPU - Kubernetes system CPU reservation - 250m), rounded down to the nearest 100m Actual memory request/limit = (nominal pod memory - Kubernetes system memory reservation - 512Mi), rounded down to the nearest 100Mi ``` | `podSize` | Nominal CPU request | Actual CPU request | Nominal memory request/limit | Actual memory request/limit | | --------- | ------------------- | ------------------ | ---------------------------- | --------------------------- | | `large` | 2 | 1600m | 8Gi | 5700Mi | | `xlarge` | 4 | 3600m | 16Gi | 13100Mi | | `2xlarge` | 8 | 7600m | 32Gi | 28500Mi | | `4xlarge` | 16 | 15600m | 64Gi | 59300Mi | | `6xlarge` | 24 | 23600m | 96Gi | 90100Mi | | `8xlarge` | 32 | 31600m | 128Gi | 120900Mi | The presets do not set a CPU limit, allowing a pod to use idle CPU on its node without being throttled. Memory requests and limits are equal to keep memory usage within the allocatable node capacity. Values defining the ingest queue sizes and search cache sizes are automatically applied for the selected tier. See the [Helm chart sizing map](https://github.com/DataDog/helm-charts/blob/main/charts/cloudprem/sizing-map.yaml) for the complete configuration. For more details on each parameter, see the Quickwit documentation for [indexer parameters](https://quickwit.io/docs/configuration/node-config#indexer-configuration), [ingest API parameters](https://quickwit.io/docs/configuration/node-config#ingest-api-configuration), and [searcher parameters](https://quickwit.io/docs/configuration/node-config#searcher-configuration). ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Configure BYOC Logs Ingress](https://docs.datadoghq.com/byoc-logs/configure/ingress.md) - [Configure BYOC Logs Log Processing](https://docs.datadoghq.com/byoc-logs/configure/pipelines.md) - [Learn more about BYOC Logs Architecture](https://docs.datadoghq.com/byoc-logs/introduction/architecture.md)