---
title: Data Observability Monitor
description: >-
  Monitor freshness, row count, column-level metrics, and custom SQL queries
  across your data warehouses.
breadcrumbs: Docs > Monitors > Monitor Types > Data Observability Monitor
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Data Observability Monitor

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

[Data Observability](https://docs.datadoghq.com/data_observability.md) monitors use anomaly detection that learns from seasonality, trends, and user feedback to catch delayed data, incomplete loads, and unexpected value changes before they affect downstream dashboards, AI applications, or business decisions. Combined with end-to-end data and code lineage, these monitors help teams detect issues early, assess downstream impact, and route to the right owner.

Data Observability monitors support the following metric types:

**Table-level metric types:**

| Metric type | Description                                             |
| ----------- | ------------------------------------------------------- |
| Freshness   | Tracks the time elapsed since a table was last updated. |
| Row Count   | Tracks the number of rows in a table or view.           |
| Custom SQL  | Tracks a custom metric value returned by a SQL query.   |

**Column-level metric types:**

| Metric type                                 | Description                                            |
| ------------------------------------------- | ------------------------------------------------------ |
| Freshness                                   | Tracks the most recent date seen in a datetime column. |
| Uniqueness                                  | Tracks the percentage of unique values.                |
| Nullness                                    | Tracks the percentage of null values.                  |
| Cardinality                                 | Tracks the number of distinct values.                  |
| Percent Zero                                | Tracks the percentage of values equal to zero.         |
| Percent Negative                            | Tracks the percentage of negative values.              |
| Min / Max / Mean / Sum / Standard Deviation | Tracks statistical measures across column values.      |

Datadog collects metrics such as row count and freshness from warehouse system metadata (for example, `INFORMATION_SCHEMA`) when available. This avoids running a query against your warehouse and reduces compute costs. Not all warehouses expose system metadata. For metrics that cannot be collected from system metadata, the monitor runs a query directly against your warehouse to compute the value.

Data Observability monitors require [Quality Monitoring](https://docs.datadoghq.com/data_observability/quality_monitoring.md) to be set up with at least one supported data warehouse (for example, [Snowflake](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/snowflake.md), [Databricks](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/databricks.md), or [BigQuery](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/bigquery.md)).

Data Observability offers four monitor types, selected in the first step of the [monitor creation flow](https://app.datadoghq.com/monitors/create/data-quality):

| Monitor type     | What it monitors                                                             |
| ---------------- | ---------------------------------------------------------------------------- |
| Data quality     | Freshness, row count, and column-level metrics on tables and columns.        |
| Source to target | The difference in the same metric between a source asset and a target asset. |
| Schema change    | Fields added, removed, renamed, or retyped in your warehouse.                |
| Job              | Failed job runs.                                                             |

Unless stated otherwise, the sections below describe the data quality monitor type.

## Monitor creation{% #monitor-creation %}

To create a Data Observability monitor in Datadog, navigate to [Data Observability > Monitors > New Monitor](https://app.datadoghq.com/monitors/create/data-quality) or [Monitors > New Monitor > Data Observability](https://app.datadoghq.com/monitors/create/data-quality). To view all existing Data Observability monitors, see the [Data Observability Monitors page](https://app.datadoghq.com/data-obs/monitors).

## Choose data to monitor{% #choose-data-to-monitor %}

First, select whether to monitor the Table or Column level:

{% image
   source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/entity_type_selection_and_aastra.9d03b241f22cfc4058a091c594a08a02.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/entity_type_selection_and_aastra.9d03b241f22cfc4058a091c594a08a02.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Choose data to monitor: entity type selector, query input, and lineage relationship filter" /%}

Then, use the Edit tab to search for tables, views, or columns by typing `key:value` filters into the search field.

**Filter by name or location:**

| Filter   | Example                 | Description                            |
| -------- | ----------------------- | -------------------------------------- |
| Name     | `name:USERS*`           | Match by name. Supports `*` wildcards. |
| Schema   | `schema:PROD`           | Match by schema.                       |
| Database | `database:ANALYTICS_DB` | Match by database.                     |
| Account  | `account:my_account`    | Match by account.                      |

**Filter by tag:**

Filter on any tag applied to your data assets by using the tag key as the filter key. For example, if your assets are tagged with `owner`, `platform`, or `environment`, search on those tags directly:

| Example                    | Description                                          |
| -------------------------- | ---------------------------------------------------- |
| `owner:data-platform-team` | Match assets tagged with `owner:data-platform-team`. |
| `platform:snowflake`       | Match assets tagged with `platform:snowflake`.       |
| `environment:production`   | Match assets tagged with `environment:production`.   |

Tag filters support the same `*` wildcards and quoting as name filters, for example, `owner:data-*` or `platform:"Snowflake Prod"`.

**Filter by computed attribute:**

In addition to your own tags, Datadog computes attributes for your data assets that you can filter on. The available computed attribute is:

| Attribute       | Values                                                    | Description                                                                                                                                                                                                                                                       |
| --------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lineage_score` | `0.00`, `0.10`, `0.30`, `0.50`, `0.70`, `0.90`, or `1.00` | A relative measure of how connected an asset is in your lineage graph, based on how many downstream assets depend on it compared to other assets of the same type. Higher values identify the tables, views, and columns that the downstream consumers depend on. |

`lineage_score` is bucketed into the discrete tiers listed above rather than taking a continuous value, so filter on one of those exact values. Match a single tier, or combine tiers with `OR`. For example, `lineage_score:1.00` returns your most depended-on assets, and `lineage_score:(0.90 OR 1.00)` returns the top two tiers.

Combine any of these filters with `AND` or `OR`, use parentheses to group conditions, and prefix with `-` to exclude.

**Examples:**

| Goal                                                     | Query                                                       |
| -------------------------------------------------------- | ----------------------------------------------------------- |
| All tables in the PROD schema, excluding temp tables     | `schema:PROD AND -name:TEMP*`                               |
| All timestamp columns                                    | `name:*_AT OR name:*_TIMESTAMP`                             |
| Tables in either PROD or STAGING for a specific database | `database:ANALYTICS_DB AND (schema:PROD OR schema:STAGING)` |
| Tables owned by a specific team                          | `owner:data-platform-team`                                  |
| The most depended-on tables in a database                | `database:ANALYTICS_DB AND lineage_score:1.00`              |

**Filter by lineage relationship:**

To scope your selection to assets that are connected to another asset in your lineage graph, click Add Relation Filter. Choose Upstream of or Downstream of, then select a specific asset or use the same `key:value` filters to match a set of assets. For example, monitor every table that is upstream of a critical dashboard, or every column downstream of a specific source table.

**Filter by hierarchy relationship:**

To scope your selection to assets that are a parent or child of another asset in your lineage graph, click Add Relation Filter. Choose Parent of or Child of, then select a specific asset or use the same `key:value` filters to match a set of assets. For example, monitor every table that has a `revenue` column, or every table that is within a critical schema.

A single monitor can track up to 5,000 tables, views, or columns. This limit cannot be increased. If your query matches more, split them across multiple monitors.

## Select your metric type{% #select-your-metric-type %}

Choose a metric type based on the data quality signal you want to track. Each monitor tracks one metric type.

{% tab title="Freshness" %}
The Freshness metric type detects when data has not been updated within an expected time window. Use it to catch stale data before it affects downstream reports or models.

- **Table freshness** tracks the time elapsed since the table was last updated. Table freshness is not available for views or for data warehouses that do not provide updated timestamps for tables in system metadata. Use column-level freshness instead.
- **Column freshness** tracks the most recent date seen in a datetime column.

{% /tab %}

{% tab title="Row Count" %}
The Row Count metric type tracks row count changes in your tables. Use it to detect unexpected drops or spikes in data that could indicate pipeline failures or upstream issues.
{% /tab %}

{% tab title="Column Metric" %}
Column metric types track column-level metrics to detect data drift or quality degradation. Select from the following:

| Metric             | Description                                                  |
| ------------------ | ------------------------------------------------------------ |
| Uniqueness         | The percentage of values in a column that are unique.        |
| Nullness           | The percentage of values in a column that are null.          |
| Cardinality        | The number of distinct values in a column.                   |
| Percent Zero       | The percentage of values in a column that are equal to zero. |
| Percent Negative   | The percentage of values in a column that are negative.      |
| Min                | The minimum of all values in a column.                       |
| Max                | The maximum of all values in a column.                       |
| Mean               | The average of all values in a column.                       |
| Standard Deviation | The measure of variation within values in a column.          |
| Sum                | The sum of all values in a column.                           |

{% alert level="info" %}
Some column metrics are only available for specific column types. Numeric metrics (Percent Zero, Percent Negative, Min, Max, Mean, Standard Deviation, Sum) require numeric columns.
{% /alert %}

{% /tab %}

{% tab title="Custom SQL" %}
The Custom SQL metric type tracks a custom metric value returned by a SQL query that you define. Use it when built-in metric types do not cover your use case, such as monitoring business-specific data quality rules.

1. Select a model type that describes the value returned by your query:
   - Default: The query returns a scalar value. Use this in most cases.
   - Freshness: The query returns the difference (in seconds) between the current time and the last time an event occurred.
   - Percentage: The query returns a percentage value between 0 and 100.
1. Write a SQL query that returns a single value aliased as `dd_value`, for example: `SELECT COUNT(*) as dd_value FROM ANALYTICS_DB.PROD.ORDERS WHERE STATUS = 'FAILED'`
1. Click Validate to verify your query syntax.

If your SQL query includes a `GROUP BY` clause, list the grouped columns as a comma-separated list in the Group by field (for example, `column_a, column_b`). Each group is evaluated independently.

**Note**: Each Custom SQL monitor counts as an individual monitored table for billing purposes.

{% image
   source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/custom_sql_example.def9e4decd735a9f1f69cf086eedf358.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/custom_sql_example.def9e4decd735a9f1f69cf086eedf358.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Input field for custom SQL monitor creation." /%}

{% /tab %}

## Configure monitor{% #configure-monitor %}

### Detection method{% #detection-method %}

Select a detection method:

- Anomalies: Alert when the metric deviates from an expected pattern. Threshold values are not required. The anomaly model requires **3 to 7 days** to train (including a weekend), depending on how frequently the underlying data updates. During the training period, the monitor does not trigger alerts and is visualized in blue. After training completes, the monitor is shown in green when in a normal state and red when in an outlier state.
- Thresholds: Alert when the metric crosses a fixed value. Set the comparison operator (`above`, `above or equal to`, `below`, `below or equal to`, `equal to`, or `not equal to`) and define a Critical threshold (required) and optionally a Warning threshold. For more details, see [Configure Monitors](https://docs.datadoghq.com/monitors/configuration.md?tab=thresholdalert#thresholds).

### WHERE clause{% #where-clause %}

Add a WHERE clause to filter the data evaluated by the monitor. This is useful for monitoring specific segments of data or only recent records. For example:

- `created_at >= DATEADD(day, -7, CURRENT_TIMESTAMP())` — only monitor rows from the past week.
- `region = 'US'` — only monitor data for a specific region.

### Group by{% #group-by %}

You can add a Group by clause to split a single monitor into multiple groups, each evaluated independently. For example, grouping a row count monitor by a `REGION` column produces a separate alert for each geography.

{% image
   source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/group_by_column_selection.19e6abc4c77a12ec85d2e2689e658574.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/group_by_column_selection.19e6abc4c77a12ec85d2e2689e658574.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Input field for selecting GROUP BY dimensions." /%}

The default limit is 500 groups per monitor. To increase this limit, [contact Support](https://docs.datadoghq.com/help/).

### Model configuration{% #model-configuration %}

For monitors using the Anomalies detection method, expand Model configuration to refine how the model behaves:

| Setting                             | Description                                                                                                                 |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Alert after N consecutive anomalies | The number of consecutive failed evaluations before the monitor alerts. Configure this setting to suppress isolated spikes. |
| Minimum upper bound size            | Constrains how tightly the model tracks your data on the upper end.                                                         |
| Minimum lower bound size            | Constrains how tightly the model tracks your data on the lower end.                                                         |

In the If data is missing to evaluate dropdown menu, select what the monitor reports when no data is available for an evaluation.

### Monitor schedule{% #monitor-schedule %}

Set how often the monitor evaluates your data:

- Scheduled: The monitor runs on a fixed cadence. Under Run this monitor, select Hourly, Every 3 hours, Every 6 hours, Every 12 hours, Daily, or Custom schedule.
- Manual (Preview): The monitor runs only when triggered programmatically. Trigger these monitors using the [Data Observability API](https://docs.datadoghq.com/api/latest/data-observability.md) on a schedule to accumulate enough historical data for modeling to be useful. The UI does not support default metrics such as row counts and freshness, so this workflow applies to custom or column-level metrics.

To define your own cadence, select Custom schedule and supply a cron expression. A custom schedule can run as often as every 15 minutes. Preview times lists the next several runs in your local time zone, so you can confirm the expression before saving it.

### Set alert conditions{% #set-alert-conditions %}

Choose an aggregation type:

- Simple Alert: Send a single notification when any monitored table or column meets the condition.
- Multi Alert: Send a notification for each group meeting the condition. Customize which dimensions to group by (for example, `table`, `schema`, `database`) to control alert granularity. For example, grouping by `schema` only sends one alert per schema, bundling all affected tables together to reduce noise.

### Example notification{% #example-notification %}

{% tab title="Threshold" %}

```text
{{#is_alert}}
Data quality issue detected on {{database.name}}.{{schema.name}}.{{table.name}}:
current value {{value}} has breached the threshold of {{threshold}}.
{{/is_alert}}

{{#is_recovery}}
Data quality issue on {{database.name}}.{{schema.name}}.{{table.name}} has recovered.
Current value {{value}} is within the threshold of {{threshold}}.
{{/is_recovery}}
```

{% /tab %}

{% tab title="Anomaly" %}

```text
{{#is_alert}}
Anomaly detected on {{database.name}}.{{schema.name}}.{{table.name}}:
observed value {{observed}} is outside the expected range of {{lower_bound}} to {{upper_bound}}
(predicted: {{predicted}}).
{{/is_alert}}

{{#is_recovery}}
{{database.name}}.{{schema.name}}.{{table.name}} has recovered.
Observed value {{observed}} is within the expected range.
{{/is_recovery}}
```

{% /tab %}

## Source to target monitors{% #source-to-target-monitors %}

{% alert level="info" %}
Source to target monitors are in Preview. Contact your Datadog representative or [support](https://docs.datadoghq.com/help/) to request access.
{% /alert %}

A source to target monitor compares the same metric on two data assets and alerts when the two values diverge. Other Data Observability monitors track whether a single asset is fresh or complete. A source to target monitor tracks whether the copy that landed in a destination matches what left the source.

When a pipeline moves data between systems, partial failures rarely look like failures. If 100,000 rows leave a source table and 99,850 rows arrive in the destination, a row count monitor on the destination alone sees a plausible value. Comparing the two assets surfaces the gap.

Use a source to target monitor to:

- Validate replication from Postgres into Databricks.
- Reconcile two databases inside the same Snowflake account, for example a quality database against production.
- Verify a migration from Redshift to BigQuery before cutover, by running both systems side by side and confirming that they match.
- Confirm that a transformation does not drop rows between its input and its output.

Source to target monitors are available in all regions except GovCloud.

### Create a source to target monitor{% #create-a-source-to-target-monitor %}

1. Navigate to [Monitors > New Monitor](https://app.datadoghq.com/monitors/create/data-quality) and select Source to Target.
1. Under Choose source, select the warehouse that holds the source data, then select the data to compare.
1. Under Choose target, do the same for the destination. The source and the target can be in different data warehouses or in the same one.
1. Under Select your metric type, choose the metric to compare. Source to target monitors support the same metric types as other Data Observability monitors, including row count, freshness, nullness, uniqueness, cardinality, and Custom SQL.
1. Set Format to control how the comparison is expressed:
   - Difference: the target value minus the source value. A negative value means the target has less than the source.
   - % Difference: the same difference as a percentage of the source value.
1. Configure the detection method, schedule, and notifications as described in Configure monitor.

The Preview Monitor Evaluation panel shows the identified source and target, along with a preview of the selected metric.

The monitored asset is the target, so the monitor appears on the target's status page.

### Compare a custom metric{% #compare-a-custom-metric %}

When the metric type is Custom SQL, supply one query for the source and one query for the target. A WHERE clause is not accepted for this metric type. Include any filtering in each query.

### Evaluation{% #evaluation %}

The difference between the source and the target is recorded as its own metric, so a source to target monitor is evaluated by the same detection methods as any other Data Observability monitor, including anomaly detection. Both sides are measured on a synchronized schedule, so the two values are captured at the same time rather than following each warehouse's default collection cadence.

## Schema change monitors{% #schema-change-monitors %}

{% alert level="info" %}
Schema change monitors are in Preview.
{% /alert %}

A schema change monitor alerts when the structure of your data changes, rather than when its contents change. Use it to catch an upstream change before it breaks a downstream pipeline or dashboard, such as when a column is dropped, renamed, or switched to a different data type.

Schema change monitors detect four kinds of change across databases, schemas, tables, and columns:

| Change type  | Description                                                       |
| ------------ | ----------------------------------------------------------------- |
| Added        | A database, schema, table, or column was created.                 |
| Removed      | A database, schema, table, or column was dropped.                 |
| Renamed      | A table or column was renamed.                                    |
| Type changed | A column's data type changed, such as from `INTEGER` to `STRING`. |

Schema changes are detected for Snowflake, BigQuery, Databricks, and Redshift.

### Create a schema change monitor{% #create-a-schema-change-monitor %}

1. Navigate to [Monitors > New Monitor > Schema Change](https://app.datadoghq.com/monitors/create/schema-change).
1. Under Choose data to monitor, select the warehouse to watch.
1. Configure notifications as described in Configure monitor.

A schema change monitor does not take a metric type or a detection method because it alerts on a structural change rather than on a measured value crossing a bound.

### Browse detected schema changes{% #browse-detected-schema-changes %}

To see the changes Datadog has detected without creating a monitor, go to [Data Observability > Schema Changes](https://app.datadoghq.com/data-obs/schema-changes). Filter by platform, account, database, schema, or change type, and expand an entry to see the affected columns and their data types.

Changes are detected when Datadog next collects schema metadata from your warehouse and compares the current structure against the previously collected one.

## Example monitors{% #example-monitors %}

{% tab title="Row count drop" %}
Detect a significant decrease in row count that could indicate a pipeline failure or missing data.

1. Select Table > Row Count and choose the target table (for example, `ANALYTICS_DB.PROD.EVENTS`).
1. Select Anomalies as the detection method. The monitor triggers when the row count deviates from its historical baseline.

{% /tab %}

{% tab title="Stale table" %}
Alert when a critical table has not been updated within the expected time window.

1. Select Table > Freshness and choose the target table (for example, `ANALYTICS_DB.PROD.ORDERS`).
1. Select Thresholds as the detection method.
1. Set the Alert threshold to **6 hours** and optionally a Warning threshold at **4 hours**.

{% /tab %}

{% tab title="Null percentage spike" %}
Detect when a column's null percentage exceeds normal levels, which may indicate data ingestion issues.

1. Select Column > Nullness and choose the target column (for example, `ANALYTICS_DB.PROD.USERS.EMAIL`).
1. Select Anomalies as the detection method.

{% /tab %}

{% tab title="Rows lost between source and target" %}
Detect rows dropped between a source table and its destination after a replication or migration.

1. Select Source to Target, then choose the source table (for example, `POSTGRES_DB.PUBLIC.ORDERS`) and the target table (for example, `ANALYTICS_DB.PROD.ORDERS`).
1. Select Row Count as the metric type and set Format to Difference.
1. Select Anomalies as the detection method.

{% /tab %}

## Annotate bounds{% #annotate-bounds %}

For monitors using the **Anomaly** detection method, you can annotate bound ranges to provide feedback and improve the model over time. Unlike infrastructure metrics, data quality metrics are often business-specific, so use annotations to teach the model what behavior is normal for your data.

{% image
   source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/annotate_bounds.42e16328fdd525848c44241c1b7a7a68.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/annotate_bounds.42e16328fdd525848c44241c1b7a7a68.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Hover menu for annotating a monitor bound." /%}

On a monitor's status page, click Annotate Bounds, select a time range on the chart, and choose one of the following annotations:

| Annotation    | Description                                               |
| ------------- | --------------------------------------------------------- |
| Expected      | Expand bounds to include the marked behavior permanently. |
| Reset for now | Mark behavior as OK, but alert if it happens again.       |
| Missed alert  | Contract bounds to alert on this behavior.                |
| Ignore        | Exclude annotated data when modeling bounds.              |

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Data Observability Overview](https://docs.datadoghq.com/data_observability.md)
- [Quality Monitoring](https://docs.datadoghq.com/data_observability/quality_monitoring.md)
- [Configure your monitor notifications](https://docs.datadoghq.com/monitors/notify.md)
- [Schedule a downtime to mute a monitor](https://docs.datadoghq.com/monitors/downtimes.md)
- [Consult your monitor status](https://docs.datadoghq.com/monitors/status.md)