--- title: Data Observability Monitor description: >- Monitor freshness, row count, column-level metrics, and custom SQL queries across your data warehouses. breadcrumbs: Docs > Monitors > Monitor Types > Data Observability Monitor --- # Data Observability Monitor {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} [Data Observability](https://docs.datadoghq.com/data_observability.md) monitors use anomaly detection that learns from seasonality, trends, and user feedback to catch delayed data, incomplete loads, and unexpected value changes before they affect downstream dashboards, AI applications, or business decisions. Combined with end-to-end data and code lineage, these monitors help teams detect issues early, assess downstream impact, and route to the right owner. Data Observability monitors support the following metric types: **Table-level metric types:** | Metric type | Description | | ----------- | ------------------------------------------------------- | | Freshness | Tracks the time elapsed since a table was last updated. | | Row Count | Tracks the number of rows in a table or view. | | Custom SQL | Tracks a custom metric value returned by a SQL query. | **Column-level metric types:** | Metric type | Description | | ------------------------------------------- | ------------------------------------------------------ | | Freshness | Tracks the most recent date seen in a datetime column. | | Uniqueness | Tracks the percentage of unique values. | | Nullness | Tracks the percentage of null values. | | Cardinality | Tracks the number of distinct values. | | Percent Zero | Tracks the percentage of values equal to zero. | | Percent Negative | Tracks the percentage of negative values. | | Min / Max / Mean / Sum / Standard Deviation | Tracks statistical measures across column values. | Datadog collects metrics such as row count and freshness from warehouse system metadata (for example, `INFORMATION_SCHEMA`) when available. This avoids running a query against your warehouse and reduces compute costs. Not all warehouses expose system metadata. For metrics that cannot be collected from system metadata, the monitor runs a query directly against your warehouse to compute the value. Data Observability monitors require [Quality Monitoring](https://docs.datadoghq.com/data_observability/quality_monitoring.md) to be set up with at least one supported data warehouse (for example, [Snowflake](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/snowflake.md), [Databricks](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/databricks.md), or [BigQuery](https://docs.datadoghq.com/data_observability/quality_monitoring/data_warehouses/bigquery.md)). ## Monitor creation{% #monitor-creation %} To create a Data Observability monitor in Datadog, navigate to [Data Observability > Monitors > New Monitor](https://app.datadoghq.com/monitors/create/data-quality) or [Monitors > New Monitor > Data Observability](https://app.datadoghq.com/monitors/create/data-quality). To view all existing Data Observability monitors, see the [Data Observability Monitors page](https://app.datadoghq.com/data-obs/monitors). ## Choose data to monitor{% #choose-data-to-monitor %} First, select whether to monitor the Table or Column level: {% image source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/entity_type_selection_and_aastra.9d03b241f22cfc4058a091c594a08a02.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/entity_type_selection_and_aastra.9d03b241f22cfc4058a091c594a08a02.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Choose data to monitor: entity type selector, query input, and lineage relationship filter" /%} Then, use the Edit tab to search for tables, views, or columns by typing `key:value` filters into the search field. **Filter by name or location:** | Filter | Example | Description | | -------- | ----------------------- | -------------------------------------- | | Name | `name:USERS*` | Match by name. Supports `*` wildcards. | | Schema | `schema:PROD` | Match by schema. | | Database | `database:ANALYTICS_DB` | Match by database. | | Account | `account:my_account` | Match by account. | **Filter by tag:** Filter on any tag applied to your data assets by using the tag key as the filter key. For example, if your assets are tagged with `owner`, `platform`, or `environment`, search on those tags directly: | Example | Description | | -------------------------- | ---------------------------------------------------- | | `owner:data-platform-team` | Match assets tagged with `owner:data-platform-team`. | | `platform:snowflake` | Match assets tagged with `platform:snowflake`. | | `environment:production` | Match assets tagged with `environment:production`. | Tag filters support the same `*` wildcards and quoting as name filters, for example, `owner:data-*` or `platform:"Snowflake Prod"`. **Filter by computed attribute:** In addition to your own tags, Datadog computes attributes for your data assets that you can filter on. The available computed attribute is: | Attribute | Values | Description | | --------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `lineage_score` | `0.00`, `0.10`, `0.30`, `0.50`, `0.70`, `0.90`, or `1.00` | A relative measure of how connected an asset is in your lineage graph, based on how many downstream assets depend on it compared to other assets of the same type. Higher values identify the tables, views, and columns that the downstream consumers depend on. | `lineage_score` is bucketed into the discrete tiers listed above rather than taking a continuous value, so filter on one of those exact values. Match a single tier, or combine tiers with `OR`. For example, `lineage_score:1.00` returns your most depended-on assets, and `lineage_score:(0.90 OR 1.00)` returns the top two tiers. Combine any of these filters with `AND` or `OR`, use parentheses to group conditions, and prefix with `-` to exclude. **Examples:** | Goal | Query | | -------------------------------------------------------- | ----------------------------------------------------------- | | All tables in the PROD schema, excluding temp tables | `schema:PROD AND -name:TEMP*` | | All timestamp columns | `name:*_AT OR name:*_TIMESTAMP` | | Tables in either PROD or STAGING for a specific database | `database:ANALYTICS_DB AND (schema:PROD OR schema:STAGING)` | | Tables owned by a specific team | `owner:data-platform-team` | | The most depended-on tables in a database | `database:ANALYTICS_DB AND lineage_score:1.00` | **Filter by lineage relationship:** To scope your selection to assets that are connected to another asset in your lineage graph, click Add Lineage Filter. Choose Upstream of or Downstream of, then select a specific asset or use the same `key:value` filters to match a set of assets. For example, monitor every table that is upstream of a critical dashboard, or every column downstream of a specific source table. A single monitor can track up to 5,000 tables, views, or columns. This limit cannot be increased. If your query matches more, split them across multiple monitors. ## Select your metric type{% #select-your-metric-type %} Choose a metric type based on the data quality signal you want to track. Each monitor tracks one metric type. {% tab title="Freshness" %} The Freshness metric type detects when data has not been updated within an expected time window. Use it to catch stale data before it affects downstream reports or models. - **Table freshness** tracks the time elapsed since the table was last updated. Table freshness is not available for views or for data warehouses that do not provide updated timestamps for tables in system metadata. Use column-level freshness instead. - **Column freshness** tracks the most recent date seen in a datetime column. {% /tab %} {% tab title="Row Count" %} The Row Count metric type tracks row count changes in your tables. Use it to detect unexpected drops or spikes in data that could indicate pipeline failures or upstream issues. {% /tab %} {% tab title="Column Metric" %} Column metric types track column-level metrics to detect data drift or quality degradation. Select from the following: | Metric | Description | | ------------------ | ------------------------------------------------------------ | | Uniqueness | The percentage of values in a column that are unique. | | Nullness | The percentage of values in a column that are null. | | Cardinality | The number of distinct values in a column. | | Percent Zero | The percentage of values in a column that are equal to zero. | | Percent Negative | The percentage of values in a column that are negative. | | Min | The minimum of all values in a column. | | Max | The maximum of all values in a column. | | Mean | The average of all values in a column. | | Standard Deviation | The measure of variation within values in a column. | | Sum | The sum of all values in a column. | {% alert level="info" %} Some column metrics are only available for specific column types. Numeric metrics (Percent Zero, Percent Negative, Min, Max, Mean, Standard Deviation, Sum) require numeric columns. {% /alert %} {% /tab %} {% tab title="Custom SQL" %} The Custom SQL metric type tracks a custom metric value returned by a SQL query that you define. Use it when built-in metric types do not cover your use case, such as monitoring business-specific data quality rules. 1. Select a model type that describes the value returned by your query: - Default: The query returns a scalar value. Use this in most cases. - Freshness: The query returns the difference (in seconds) between the current time and the last time an event occurred. - Percentage: The query returns a percentage value between 0 and 100. 1. Write a SQL query that returns a single value aliased as `dd_value`, for example: `SELECT COUNT(*) as dd_value FROM ANALYTICS_DB.PROD.ORDERS WHERE STATUS = 'FAILED'` 1. Click Validate to verify your query syntax. If your SQL query includes a `GROUP BY` clause, list the grouped columns as a comma-separated list in the Group by field (for example, `column_a, column_b`). Each group is evaluated independently. **Note**: Each Custom SQL monitor counts as an individual monitored table for billing purposes. {% image source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/custom_sql_example.def9e4decd735a9f1f69cf086eedf358.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/custom_sql_example.def9e4decd735a9f1f69cf086eedf358.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Input field for custom SQL monitor creation." /%} {% /tab %} ## Configure monitor{% #configure-monitor %} ### Detection method{% #detection-method %} Select a detection method: - Anomaly: Alert when the metric deviates from an expected pattern. Threshold values are not required. The anomaly model requires **3 to 7 days** to train (including a weekend), depending on how frequently the underlying data updates. During the training period, the monitor does not trigger alerts and will be visualized in blue. After training completes, the monitor will be shown in green when in a normal state and red when in an outlier state. - Threshold: Alert when the metric crosses a fixed value. Set the comparison operator (`above`, `above or equal to`, `below`, `below or equal to`, `equal to`, or `not equal to`) and define a Critical threshold (required) and optionally a Warning threshold. For more details, see [Configure Monitors](https://docs.datadoghq.com/monitors/configuration.md?tab=thresholdalert#thresholds). ### WHERE clause{% #where-clause %} Add a WHERE clause to filter the data evaluated by the monitor. This is useful for monitoring specific segments of data or only recent records. For example: - `created_at >= DATEADD(day, -7, CURRENT_TIMESTAMP())` — only monitor rows from the past week. - `region = 'US'` — only monitor data for a specific region. ### Group by{% #group-by %} You can add a Group by clause to split a single monitor into multiple groups, each evaluated independently. For example, grouping a row count monitor by a `REGION` column produces a separate alert for each geography. {% image source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/group_by_column_selection.19e6abc4c77a12ec85d2e2689e658574.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/group_by_column_selection.19e6abc4c77a12ec85d2e2689e658574.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Input field for selecting GROUP BY dimensions." /%} The default limit is 500 groups per monitor. To increase this limit, [contact Support](https://docs.datadoghq.com/help/). ### Monitor schedule{% #monitor-schedule %} Set how often the monitor evaluates your data: - Hourly: The monitor runs every hour. - Daily: The monitor runs once per day. - Manual: The monitor runs only when triggered programmatically. Trigger these monitors using the [Data Observability API](https://docs.datadoghq.com/api/latest/data-observability.md) on a schedule so enough historical data can accumulate for modeling to be useful. Currently, the UI does not support default metrics like row counts and freshness, so this workflow only applies to custom or column-level metrics. ### Set alert conditions{% #set-alert-conditions %} Choose an aggregation type: - Simple Alert: Send a single notification when any monitored table or column meets the condition. - Multi Alert: Send a notification for each group meeting the condition. Customize which dimensions to group by (for example, `table`, `schema`, `database`) to control alert granularity. For example, grouping by `schema` only sends one alert per schema, bundling all affected tables together to reduce noise. ### Example notification{% #example-notification %} {% tab title="Threshold" %} ```text {{#is_alert}} Data quality issue detected on {{database.name}}.{{schema.name}}.{{table.name}}: current value {{value}} has breached the threshold of {{threshold}}. {{/is_alert}} {{#is_recovery}} Data quality issue on {{database.name}}.{{schema.name}}.{{table.name}} has recovered. Current value {{value}} is within the threshold of {{threshold}}. {{/is_recovery}} ``` {% /tab %} {% tab title="Anomaly" %} ```text {{#is_alert}} Anomaly detected on {{database.name}}.{{schema.name}}.{{table.name}}: observed value {{observed}} is outside the expected range of {{lower_bound}} to {{upper_bound}} (predicted: {{predicted}}). {{/is_alert}} {{#is_recovery}} {{database.name}}.{{schema.name}}.{{table.name}} has recovered. Observed value {{observed}} is within the expected range. {{/is_recovery}} ``` {% /tab %} ## Example monitors{% #example-monitors %} {% tab title="Row count drop" %} Detect a significant decrease in row count that could indicate a pipeline failure or missing data. 1. Select Table > Row Count and choose the target table (for example, `ANALYTICS_DB.PROD.EVENTS`). 1. Select Anomaly as the detection method. The monitor triggers when the row count deviates from its historical baseline. {% /tab %} {% tab title="Stale table" %} Alert when a critical table has not been updated within the expected time window. 1. Select Table > Freshness and choose the target table (for example, `ANALYTICS_DB.PROD.ORDERS`). 1. Select Threshold as the detection method. 1. Set the Alert threshold to **6 hours** and optionally a Warning threshold at **4 hours**. {% /tab %} {% tab title="Null percentage spike" %} Detect when a column's null percentage exceeds normal levels, which may indicate data ingestion issues. 1. Select Column > Nullness and choose the target column (for example, `ANALYTICS_DB.PROD.USERS.EMAIL`). 1. Select Anomaly as the detection method. {% /tab %} ## Annotate bounds{% #annotate-bounds %} For monitors using the **Anomaly** detection method, you can annotate bound ranges to provide feedback and improve the model over time. Unlike infrastructure metrics, data quality metrics are often business-specific, so use annotations to teach the model what behavior is normal for your data. {% image source="https://docs.dd-static.net/images/monitors/monitor_types/data_observability/annotate_bounds.42e16328fdd525848c44241c1b7a7a68.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/monitors/monitor_types/data_observability/annotate_bounds.42e16328fdd525848c44241c1b7a7a68.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Hover menu for annotating a monitor bound." /%} On a monitor's status page, click Annotate Bounds, select a time range on the chart, and choose one of the following annotations: | Annotation | Description | | ------------- | --------------------------------------------------------- | | Expected | Expand bounds to include the marked behavior permanently. | | Reset for now | Mark behavior as OK, but alert if it happens again. | | Missed alert | Contract bounds to alert on this behavior. | | Ignore | Exclude annotated data when modeling bounds. | ## Further Reading{% #further-reading %} - [Data Observability Overview](https://docs.datadoghq.com/data_observability.md) - [Quality Monitoring](https://docs.datadoghq.com/data_observability/quality_monitoring.md) - [Configure your monitor notifications](https://docs.datadoghq.com/monitors/notify.md) - [Schedule a downtime to mute a monitor](https://docs.datadoghq.com/monitors/downtimes.md) - [Consult your monitor status](https://docs.datadoghq.com/monitors/status.md)