--- title: Data Observability Integration Overhead description: >- Understand the compute overhead Data Observability adds when monitoring data quality and jobs, and how to control it. breadcrumbs: Docs > Data Observability Overview > Data Observability Integration Overhead --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Data Observability Integration Overhead {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} Data Observability monitors your data through two products, each with a different overhead profile: - **Quality Monitoring** evaluates data-quality metrics (such as freshness, row count, and column statistics) on a schedule. Depending on the warehouse and metric, an evaluation either reads table metadata or runs a SQL query against your data, so its cost is measured in **warehouse compute** (for example, Snowflake credits, Databricks DBUs, or BigQuery bytes scanned). - **Jobs Monitoring** observes the performance of your jobs by running the **Datadog Agent** on your compute (for example, Databricks or Spark clusters). The Agent shares a small amount of the CPU and memory you already provision for your workloads. This page describes both overhead sources and how to control them. Actual cost depends heavily on your environment: warehouse size, the amount of data scanned, how often monitors run, and warehouse auto-stop settings. This page focuses on what consumes resources, not a single benchmark number. ## Quality Monitoring overhead{% #quality-monitoring-overhead %} A Data Observability monitor evaluates its metric on a fixed schedule that you choose: **hourly** or **daily**. Each time a monitor runs, Data Observability collects the metric value, either by reading table metadata or by running a query against your data. How each metric is collected depends on the warehouse. {% tab title="Snowflake" %} On Snowflake, **row count** and **table freshness** are read from table metadata rather than by scanning the table. Datadog reads the `ROW_COUNT` and `LAST_ALTERED` columns from `INFORMATION_SCHEMA.TABLES`, and batches many tables in a database into a single query. This reads metadata only and does not scan table data. However, querying `INFORMATION_SCHEMA` requires a running warehouse, so it uses a small, roughly fixed amount of warehouse compute per batch, regardless of table size. **Column metrics** (Nullness, Uniqueness, Cardinality, Percent Zero, Percent Negative, Min, Max, Mean, Sum, Standard Deviation, and column-level freshness) and **Custom SQL** run as `SELECT` queries against your tables and are billed as normal warehouse usage. Column metrics on the same table that share the same filter and grouping are combined into a single query. {% /tab %} {% tab title="Databricks" %} On Databricks, every metric is collected by running a SQL statement on the SQL warehouse you connect for Quality Monitoring. There is no separate no-cost metadata layer for row count or freshness; these values are computed by querying the warehouse. As a result, each evaluation requires that SQL warehouse to be active and consumes DBUs while it runs. | Metric | Statement run on the SQL warehouse | Notes | | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | **Row count** | `SELECT COUNT(1) FROM ` | An aggregate query against the table. The data actually scanned depends on table size and Delta optimizations such as data skipping and statistics. | | **Table freshness** (base table) | `SELECT timestamp FROM (DESCRIBE HISTORY
LIMIT 1)` | A lightweight metadata command—scans little data, but still runs on the warehouse, so the warehouse must be active. | | **Table freshness** (materialized view) | `DESCRIBE TABLE EXTENDED
` | Lightweight metadata command. | | **Table freshness** (standard view) | *(no query)* | Views have no freshness query and add no overhead; use column-level freshness instead. | | **Column metrics** (Nullness, Uniqueness, Cardinality, Percent Zero, Percent Negative, Min, Max, Mean, Sum, Standard Deviation, column freshness) | Aggregate `SELECT` over the column | An aggregate query against the table. Column metrics on the same table that share the same filter and grouping are combined into a single query. | | **Custom SQL** | Your query, run as written | Cost is whatever your query costs to run. | {% /tab %} {% tab title="BigQuery" %} On BigQuery, **row count** and **table freshness** are read through BigQuery's table metadata API (equivalent to a `tables.get` call). No query runs and no bytes are scanned, so these checks incur no BigQuery query cost. **Column metrics** (Nullness, Uniqueness, Cardinality, Percent Zero, Percent Negative, Min, Max, Mean, Sum, Standard Deviation, and column-level freshness) and **Custom SQL** run as BigQuery SQL against your tables and are billed by the data they process—bytes scanned under on-demand pricing, or slot time under capacity pricing. Column metrics on the same table that share the same filter and grouping are combined into a single query. {% /tab %} ### What drives the cost{% #what-drives-the-cost %} Warehouse compute consumed by Quality Monitoring scales with: - **Cadence**: an hourly monitor reruns its query 24 times as often as a daily monitor. - **Number of distinct queries**: cost scales with the number of distinct *(table, filter, grouping)* combinations being monitored, not the raw number of metrics. Multiple column metrics on the same table are batched into one query. - **Data scanned per query**: column-statistic queries (and Custom SQL) are aggregates that scan table data, so larger tables and partitions cost more on every warehouse. Row count scans the table only on Databricks (`COUNT`); on Snowflake and BigQuery it is a metadata read that does not grow with table size. Freshness checks are lightweight everywhere: metadata reads on Snowflake and BigQuery, and metadata commands on Databricks. - **Warehouse warm time**: on warehouse-based platforms (Snowflake, Databricks), the dominant factor on most bills is how long your warehouse stays running. Frequent checks against a warehouse with a long idle timeout keep it warm and accrue cost even between queries. BigQuery on-demand is serverless and bills only by bytes scanned, so it has no warm-time cost. Data Observability runs these queries with bounded concurrency and a per-query timeout. A backlog of monitors does not flood your warehouse with unbounded parallel queries. ### Reducing Quality Monitoring overhead{% #reducing-quality-monitoring-overhead %} - **Choose daily over hourly** for metrics that do not need hourly resolution. - **Monitor the tables and columns that matter.** Cost scales with the number of distinct queries; focus on critical tables rather than monitoring everything. - **Add a `WHERE` clause** to scope a monitor to recent partitions or a relevant data segment, reducing the data scanned. - **Group column metrics on the same table** so they batch into a single query. - **On Snowflake and Databricks, use a dedicated, right-sized warehouse with an aggressive auto-stop** so it shuts down promptly between checks. This is usually the most impactful optimization. - **Prefer table freshness on base tables** where it is a metadata read or lightweight command rather than a full aggregate scan. ## Jobs Monitoring overhead{% #jobs-monitoring-overhead %} Jobs Monitoring collects telemetry in one of two ways, each with a different overhead profile: - **Datadog Agent**: for Spark jobs on Databricks, Amazon EMR, Google Dataproc, and Kubernetes, you install the Datadog Agent on your compute. Its footprint is described in this section. - **OpenLineage events**: for Apache Airflow, dbt, and custom pipelines, you run an open source OpenLineage integration in your own environment, which emits events to Datadog. See OpenLineage-based integrations. On Databricks **classic** clusters (all-purpose or job clusters), the Agent is installed on the cluster nodes through an init script. On **serverless** compute, no Agent installation is required. ### What runs on the cluster{% #what-runs-on-the-cluster %} - The **Datadog Agent** and the Java APM tracer run as processes on the cluster nodes (the driver and the executors). - The Spark integration runs **in-process** as a Spark listener: it reacts to Spark scheduler events (job, stage, and task callbacks) to build traces and spans. It does not poll your data or run any queries against your warehouse. - The Agent collects Spark performance metrics and cluster system metrics, and optionally tails driver and worker logs. ### Footprint{% #footprint %} The Agent is a lightweight process that shares the CPU and memory you already provision for your workloads. It does not add separate, Datadog-provisioned compute to classic clusters. Jobs Monitoring traces are coarse-grained. The Spark integration emits one span per application, job, stage, SQL execution, and streaming micro-batch, plus one span for each *failed* task. Successful tasks are aggregated into stage-level metrics instead of emitting individual spans. As a result, trace volume scales with a job's number of stages and jobs, not its task count or data volume. Trace throughput therefore stays low for typical jobs. The larger contributors to the Agent's footprint are Spark metric collection and, if enabled, log collection. For the Datadog SDK's resource profile, see [APM Agent resource usage](https://docs.datadoghq.com/tracing/troubleshooting/agent_apm_resource_usage.md). The cluster Agent issues no queries against your warehouse, so it adds no warehouse cost on its own. ### Reducing Jobs Monitoring overhead{% #reducing-jobs-monitoring-overhead %} - **Disable log collection** when you do not need driver and worker logs (`DRIVER_LOGS_ENABLED` / `WORKER_LOGS_ENABLED`), or filter logs with `DD_LOGS_CONFIG_PROCESSING_RULES`. Log collection is the main tunable contributor to footprint and ingestion volume. ### Databricks cost data{% #databricks-cost-data %} Jobs Monitoring can surface the DBU cost of your Databricks jobs. This cost data is read from Datadog's own cost metrics; collecting it does not add queries to your monitored clusters. Populating Databricks cost requires the Datadog Databricks cost integration, which reads from Databricks system tables through a SQL warehouse you grant access to. For the required permissions, see the [Jobs Monitoring for Databricks setup](https://docs.datadoghq.com/data_observability/jobs_monitoring/databricks.md). ### OpenLineage-based integrations{% #openlineage-based-integrations %} Apache Airflow, dbt, and custom pipelines are monitored without the Datadog Agent. Instead, an open source [OpenLineage](https://openlineage.io/docs/integrations/about) integration emits lineage events to Datadog. These components are maintained by the OpenLineage and Apache Airflow projects, and their configuration and performance are documented in their respective project documentation. The overhead is the integration's, not the Datadog Agent's. These integrations capture run, job, and dataset metadata and emit events at run lifecycle points. The cost is proportional to the number of runs, tasks, and the metadata collected, not to your data volume. - **Airflow**: the [`apache-airflow-providers-openlineage`](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/) provider runs in your Airflow schedulers and workers and emits an event at task and DAG lifecycle points. Its [configuration reference](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/configurations-ref.html) documents options that affect overhead, such as selectively enabling OpenLineage and limiting collected metadata. - **dbt**: for dbt Core, the [`openlineage-dbt`](https://openlineage.io/docs/integrations/dbt) wrapper (`dbt-ol`) follows dbt's structured logs and emits events as the run progresses. Because it reads logs rather than instrumenting execution, it does not affect dbt's performance. dbt Cloud sends job-run events to Datadog through a webhook, with no component running in your environment. - **Custom pipelines**: you emit events yourself with the OpenLineage client or a raw HTTP call, so the overhead is whatever your emitting code does. For setup and the Datadog-optimized transport, see [Custom Jobs using OpenLineage](https://docs.datadoghq.com/data_observability/jobs_monitoring/openlineage.md). See the OpenLineage documentation for the full set of tuning and overhead controls. ## Estimating your actual costs{% #estimating-your-actual-costs %} The behaviors described on this page are environment-dependent. Actual cost depends on your warehouse size and pricing model, the data scanned, and how often monitors run. On warehouse-based platforms, it also depends on your auto-stop configuration. To estimate costs, validate against your own usage: your warehouse's query history or billing views, or Datadog Cloud Cost Management. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Data Observability Overview](https://docs.datadoghq.com/data_observability.md) - [Quality Monitoring](https://docs.datadoghq.com/data_observability/quality_monitoring.md) - [Jobs Monitoring for Databricks](https://docs.datadoghq.com/data_observability/jobs_monitoring/databricks.md) - [Data Observability Monitors](https://docs.datadoghq.com/monitors/types/data_observability.md) - [APM Agent resource usage](https://docs.datadoghq.com/tracing/troubleshooting/agent_apm_resource_usage.md)