For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/byoc-logs/introduction/architecture.md. A documentation index is available at /llms.txt.

Architecture

Join the Preview!

BYOC Logs is in Preview.

Overview

BYOC Logs architecture showing Indexers, Searchers, Metastore, and Control Plane components interacting with object storage

BYOC Logs uses a decoupled architecture which separates the compute (indexing and searching) and data on an object storage. This allows for independent scaling and optimization of different cluster components based on workload demands.

Components

The BYOC Logs cluster, typically deployed on Kubernetes (EKS), consists of several components:

Indexers
Responsible for receiving logs from Datadog Agents. Indexers process, index, and store logs in index files called splits to the object storage (for example, Amazon S3).
Searchers
Handle search queries from the Datadog UI, reading metadata from Metastore and fetching data from the object storage.
Metastore
Stores metadata about the indexes, including split locations on the object storage. BYOC Logs uses PostgreSQL for this purpose.
Control Plane
Schedules indexing jobs called indexing pipelines on indexers.
Janitor
Performs maintenance tasks, applying retention policies, garbage collecting expired splits, and running delete query jobs.

Data flow

Ingestion path (logs entering BYOC Logs)

Logs are ingested into BYOC Logs within your infrastructure. The typical flow is:

  1. Your applications emit logs to the Datadog Agent or Observability Pipelines Worker.
  2. Logs are forwarded to BYOC Logs indexers running in your cluster.
  3. Indexers process and store logs as splits in your object storage (for example, Amazon S3 or Google Cloud Storage).

No log data leaves your environment during ingestion. Logs are stored exclusively in your own object storage.

Query path (searching logs from Datadog UI)

When you search BYOC Logs data from the Datadog UI (for example, in the Log Explorer), the query flows through a secure connection between Datadog and your cluster:

  1. The Datadog UI sends the search query to Datadog’s backend.
  2. Datadog’s backend forwards the query to your BYOC Logs cluster through the established connection (reverse connection or ingress).
  3. Searchers in your cluster execute the query against your object storage.
  4. Only the matching log results are sent back to Datadog for display in the UI.

Only query results travel between your cluster and Datadog. The full dataset remains in your object storage and is never transferred to Datadog.

Connection to Datadog UI

There are two ways to connect the Datadog UI to BYOC Logs:

Further reading

Additional helpful documentation, links, and articles:

BYOC Logs Installation PrerequisitesDOCUMENTATION more more