---
title: Architecture
description: >-
  Understand how Observability Pipelines and the BYOC Logs engine collect,
  store, maintain, and query logs in your environment.
breadcrumbs: Docs > BYOC Logs > Introduction to BYOC Logs > Architecture
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Architecture

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

{% image
   source="https://docs.dd-static.net/images/cloudprem/overview_architecture_v2.08ae0729e4bba6658126c5ed74f8199f.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/cloudprem/overview_architecture_v2.08ae0729e4bba6658126c5ed74f8199f.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="BYOC Logs architecture showing log sources sending data through Observability Pipelines to the BYOC Logs engine. The engine contains indexers, compactors, searchers, a control plane, a metastore, and a janitor, and uses object storage and PostgreSQL. Searchers connect to the Datadog UI." /%}

BYOC (Bring Your Own Cloud) Logs consists of two main blocks:

- [**Observability Pipelines**](https://docs.datadoghq.com/observability_pipelines.md) collects, processes, and routes logs.
- The **BYOC Logs engine** indexes, stores, and queries logs. Datadog distributes the engine as the `datadog/cloudprem` Docker image.

The Observability Pipelines Worker and the BYOC Logs engine process log data in your environment. Datadog hosts the pipeline configuration UI and Log Explorer, and routes queries to the engine.

The BYOC Logs engine separates compute from object storage. You can scale ingestion, compaction, and search independently based on workload demand while keeping log data in your object storage.

## Architecture components{% #architecture-components %}

### Observability Pipelines{% #observability-pipelines %}

Observability Pipelines is the ingestion block for BYOC Logs. The Observability Pipelines Worker receives logs from Datadog Agents and other sources, applies the processing rules defined in your pipeline, and routes the processed logs to the BYOC Logs engine. You configure the pipeline in Datadog, and the Worker runs in your environment.

### BYOC Logs engine{% #byoc-logs-engine %}

The BYOC Logs engine runs in your environment. In Kubernetes deployments, the `datadog/cloudprem` image runs with different roles:

{% dl %}

{% dt %}
**Indexers**
{% /dt %}

{% dd %}
Receive processed logs from Observability Pipelines, create index files called *splits*, and write the splits to object storage.
{% /dd %}

{% dt %}
**Compactors**
{% /dt %}

{% dd %}
Read splits from object storage, merge them, and write the merged splits back. Compactors run on dedicated nodes to isolate merge work from indexing.
{% /dd %}

{% dt %}
**Searchers**
{% /dt %}

{% dd %}
Execute queries from Datadog, read index metadata through the metastore, and retrieve matching data from object storage.
{% /dd %}

{% dt %}
**Metastore**
{% /dt %}

{% dd %}
Provides access to index metadata, including the locations of splits in object storage. The metastore persists this metadata in PostgreSQL.
{% /dd %}

{% dt %}
**Control plane**
{% /dt %}

{% dd %}
Schedules indexing jobs called *indexing pipelines* on indexers.
{% /dd %}

{% dt %}
**Janitor**
{% /dt %}

{% dd %}
Performs maintenance tasks, applying retention policies, garbage collecting expired splits, and running delete query jobs.
{% /dd %}

{% /dl %}

### Data stores{% #data-stores %}

The engine uses two data stores in your environment:

- **Object storage** stores indexed log data as splits. Supported services include Amazon S3, Azure Blob Storage, Google Cloud Storage, and S3-compatible storage.
- **PostgreSQL** stores index metadata used by the metastore.

## Data flow{% #data-flow %}

### Ingestion and compaction path{% #ingestion-and-compaction-path %}

Logs remain in your environment throughout ingestion:

1. Applications and other sources send logs to an **Observability Pipelines Worker**.
1. The Worker processes the logs and routes them to the BYOC Logs engine.
1. **Indexers** create splits in object storage and register their metadata through the metastore.
1. **Compactors** asynchronously merge splits on dedicated nodes and write the merged splits back to object storage.

**No log data leaves your environment during ingestion.** Logs are stored exclusively in your own object storage.

### Query path{% #query-path %}

When you search BYOC Logs data in Log Explorer, the query travels over a secure connection between Datadog and the BYOC Logs engine:

1. The Datadog UI sends the search query to Datadog's backend.
1. Datadog forwards the query to the engine through a reverse connection or ingress.
1. **Searchers** use metadata from PostgreSQL to retrieve matching data from object storage.
1. The engine returns only the matching results to Datadog for display.

**Only query results travel between your environment and Datadog.** The full dataset remains in your object storage and is never transferred to Datadog.

## Connection to Datadog UI{% #connection-to-datadog-ui %}

Connect the Datadog UI to the BYOC Logs engine in one of two ways:

- **[Reverse connection](https://docs.datadoghq.com/byoc-logs/introduction/network.md)**: Let the engine initiate a secure connection to Datadog.
- **[External requests from Datadog](https://docs.datadoghq.com/byoc-logs/configure/ingress.md)**: Provide Datadog with a DNS endpoint and configure a public ingress to accept requests.

## Further reading{% #further-reading %}

- [Observability Pipelines](https://docs.datadoghq.com/observability_pipelines.md)
- [BYOC Logs Installation Prerequisites](https://docs.datadoghq.com/byoc-logs/install.md)
- [Connect BYOC Logs to Datadog](https://docs.datadoghq.com/byoc-logs/introduction/network.md)