--- title: Cluster Sizing description: Learn about cluster sizing for BYOC Logs breadcrumbs: Docs > BYOC Logs > Operate BYOC Logs > Cluster Sizing --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Cluster Sizing {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} Size your BYOC (Bring Your Own Cloud) Logs cluster in three steps: 1. Estimate your daily ingest volume in TB/day. 1. Pick a starter configuration for that volume. 1. Monitor the cluster and adjust replica counts and pod sizes. Searcher capacity depends on query concurrency, query complexity, and the amount of data scanned, not on ingest volume alone. These recommendations assume modern x86 CPUs, such as those used in AWS M6 instance types, or equivalent CPUs from other cloud providers. ARM-based CPUs, such as AWS Graviton, may provide better cost efficiency at comparable throughput. ## Starter configurations{% #starter-configurations %} Use these totals as a starting point: - **Indexers:** 2 vCPUs per TB/day - **Compactors:** 1 vCPU per 2 TB/day - **Searchers:** about twice the indexer vCPU total. Analytics-heavy workloads may need up to twice the values shown in the table. Object storage totals assume 30-day retention and a 6x compression ratio. | Daily volume | Indexers | Compactors | Searchers | Object storage | | -------------- | --------- | ---------- | --------- | -------------- | | **1 TB/day** | 2 vCPUs | 0.5 vCPUs | 4 vCPUs | ~5 TB | | **10 TB/day** | 20 vCPUs | 5 vCPUs | 40 vCPUs | ~50 TB | | **100 TB/day** | 200 vCPUs | 50 vCPUs | 400 vCPUs | ~500 TB | Recommended size for each pod: | Daily volume | Indexers | Compactors | Searchers | | ------------------- | -------------- | -------------- | ---------------- | | **Up to 30 TB/day** | 4 vCPUs, 16 GB | 4 vCPUs, 16 GB | 16 vCPUs, 64 GB | | **Above 30 TB/day** | 8 vCPUs, 32 GB | 8 vCPUs, 32 GB | 64 vCPUs, 256 GB | {% alert level="info" %} **Billing vs. provisioning:** Provisioned vCPUs and billed vCPUs are different. A production cluster is intentionally overprovisioned to absorb ingestion and search spikes. Contact your Datadog representative for billing guidance. {% /alert %} ## Size each component{% #size-each-component %} Adjust the starter configuration component by component. For the role each component plays, see [Architecture](https://docs.datadoghq.com/byoc-logs/introduction/architecture.md). ### Indexers{% #indexers %} - **Performance:** 2 vCPUs per TB/day - **Memory:** 4 GB RAM per vCPU - **Storage type:** Network-attached block storage for the write-ahead log. See [Configure persistent storage for indexers](https://docs.datadoghq.com/byoc-logs/operate/best_practices.md#configure-persistent-storage-for-indexers). {% collapsible-section %} #### Sizing by event count If you know your daily event count but not your byte volume, use this formula to estimate: $$\text"Daily volume (TB)" = {\text"events per day" × \text"average event size (bytes)"} / 10^{12}$$ For example, with 1 billion events/day at 1 KB average size: `1,000,000,000 × 1,000 / 1,000,000,000,000 = 1 TB/day` Typical log event sizes range from 500 bytes (short syslog) to 2-3 KB (JSON with Kubernetes tags). Measure a representative sample of your logs to get an accurate average. {% /collapsible-section %} ### Compactors{% #compactors %} - **Performance:** 1 vCPU per 2 TB/day - **Memory:** 4 GB RAM per vCPU - **Storage type:** Local SSD. Use instances with local SSDs, such as AWS M8gd. ### Searchers{% #searchers %} Size searchers for the expected search workload, not ingest volume alone. A starting point is about twice the indexer vCPU total. - **Performance:** Term queries (`status:error AND message:exception`) usually use less CPU than wildcard or whole-event searches. Aggregation queries need more CPU and memory. - **Memory:** 4 GB RAM per searcher vCPU. Provision more RAM if you expect many concurrent aggregation requests. If search latency is high, add searcher replicas or increase memory per pod. See [Scale searchers based on your query patterns](https://docs.datadoghq.com/byoc-logs/operate/best_practices.md#scale-searchers-based-on-your-query-patterns). ### Other services{% #other-services %} Allocate the following resources for these lightweight components: | Service | vCPUs | RAM | Replicas | | ----------------- | ----- | ---- | -------- | | **Control Plane** | 2 | 4 GB | 1 | | **Metastore** | 2 | 4 GB | 2 | | **Janitor** | 2 | 4 GB | 1 | ### PostgreSQL database{% #postgresql-database %} - **Instance size:** For most use cases, a PostgreSQL instance with 1 vCPU and 4 GB of RAM is sufficient. - **Amazon RDS recommendation:** On Amazon RDS, start with the `t4g.medium` instance type. - **High availability:** Enable Multi-AZ deployment with one standby replica. Enable automated backups on the metastore database. See [Enable automated backups on your metastore database](https://docs.datadoghq.com/byoc-logs/operate/best_practices.md#enable-automated-backups-on-your-metastore-database). ### Object storage{% #object-storage %} BYOC Logs compresses and indexes log data before storing it in object storage. Compression is typically 5x to 8x, which translates to about 125-200 GB stored per TB ingested per day. $$\text"Stored data per day" = {\text"Daily volume"} / {\text"compression ratio"}$$ $$\text"Total storage" = \text"Stored data per day" × \text"retention period (days)"$$ {% alert level="info" %} Use standard-tier object storage (for example, S3 Standard or GCS Standard) for active data. Lower-cost tiers such as S3 Infrequent Access or GCS Nearline are not validated for use with BYOC Logs. {% /alert %} To estimate PUT request volume and cost, see [Object Storage Request Estimation](https://docs.datadoghq.com/byoc-logs/operate/object_storage_requests.md). ## Helm chart sizing tiers{% #helm-chart-sizing-tiers %} Set `indexer.podSize` and `searcher.podSize` to match the per-pod CPU and memory in the starter configuration. The default is `xlarge`. Each preset also applies ingest queue and search cache sizes. | `podSize` | CPU | Memory | | --------- | --- | ------ | | `large` | 2 | 8Gi | | `xlarge` | 4 | 16Gi | | `2xlarge` | 8 | 32Gi | | `4xlarge` | 16 | 64Gi | | `6xlarge` | 24 | 96Gi | | `8xlarge` | 32 | 128Gi | {% collapsible-section %} ### Actual Kubernetes requests Each `podSize` requests less than its nominal CPU and memory, to leave room for kube-system, DaemonSets, and add-ons. The reservation amounts follow the [GKE node reservation calculation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/plan-node-sizes#resource_reservations), plus 250m CPU and 512Mi memory per node for DaemonSets and add-ons. | `podSize` | Actual CPU request | Actual memory request/limit | | --------- | ------------------ | --------------------------- | | `large` | 1600m | 5700Mi | | `xlarge` | 3600m | 13100Mi | | `2xlarge` | 7600m | 28500Mi | | `4xlarge` | 15600m | 59300Mi | | `6xlarge` | 23600m | 90100Mi | | `8xlarge` | 31600m | 120900Mi | ```text Actual CPU request = (nominal pod CPU - Kubernetes system CPU reservation - 250m), rounded down to the nearest 100m Actual memory request/limit = (nominal pod memory - Kubernetes system memory reservation - 512Mi), rounded down to the nearest 100Mi ``` {% /collapsible-section %} See the [Helm chart sizing map](https://github.com/DataDog/helm-charts/blob/main/charts/cloudprem/sizing-map.yaml) for the complete configuration. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Configure BYOC Logs Ingress](https://docs.datadoghq.com/byoc-logs/configure/ingress.md) - [Configure BYOC Logs Log Processing](https://docs.datadoghq.com/byoc-logs/configure/pipelines.md) - [Learn more about BYOC Logs Architecture](https://docs.datadoghq.com/byoc-logs/introduction/architecture.md)