--- title: Troubleshooting Monitor Alerts description: >- Troubleshoot common monitor alerting issues including state mismatches, data verification, alert conditions, and notification problems. breadcrumbs: Docs > Monitors > Monitor Guides > Troubleshooting Monitor Alerts --- # Troubleshooting Monitor Alerts ## Overview{% #overview %} This guide provides an overview of some foundational concepts that can help you determine if your monitor's alerting behavior is valid. If you suspect that your monitor's evaluations are not accurately reflecting the underlying data, use this guide to inspect your monitor and troubleshoot the following: - Monitor state and status behavior - Data availability and evaluation issues - Alert condition configuration - Monitor groups and retention - Notification issues ## Monitor state and status{% #monitor-state-and-status %} While monitor *evaluations* are stateless, meaning that the result of a given evaluation does not depend on the results of previous evaluations, monitors themselves are stateful, and their state is updated based on the evaluation results of their queries and configurations. A monitor evaluation with a given status won't necessarily cause the monitor's state to change to the same status. ### How status is tracked{% #how-status-is-tracked %} For both monitor evaluations and state, status is tracked by group. For a multi alert monitor, a group is a set of tags with one value for each grouping key (for example, `env:dev, host:myhost` for a monitor grouped by `env` and `host`). For a simple alert, there is only one group (`*`), representing everything within the monitor's scope. ### When state doesn't match evaluation{% #when-state-doesnt-match-evaluation %} The state of a monitor may also sometimes update in the absence of a monitor evaluation, for example, due to [auto-resolve](https://docs.datadoghq.com/monitors/configuration/?tabs=thresholdalert#auto-resolve). ## Data availability and evaluation issues{% #data-availability-and-evaluation-issues %} If your monitor's state or status is not what you expect, confirm the behavior of the underlying data source. For a metric monitor, you can use the [history](https://docs.datadoghq.com/monitors/status/#history) graph to view the datapoints being pulled in by the metric query. `N/A` groups are not included in monitors but are visible in dashboard queries. ### Sparse metrics{% #sparse-metrics %} If metrics are absent from a monitor's evaluation window, and the monitor is not configured to anticipate [no-data conditions](https://docs.datadoghq.com/monitors/configuration/?tabs=thresholdalert#no-data), the evaluation may be `skipped`. In such a case, the monitor state is not updated, so a monitor previously in the `OK` state remains `OK`, and likewise with a monitor in the `Alert` state. Use the [history](https://docs.datadoghq.com/monitors/status/#history) graph on the monitor status page and select the group and time frame of interest. If data is sparsely populated, see [monitor arithmetic and sparse metrics](https://docs.datadoghq.com/monitors/guide/monitor-arithmetic-and-sparse-metrics/) for more information. ### "No Data" status with rollup functions{% #no-data-status-with-rollup-functions %} If your monitors are unexpectedly evaluating in a "No Data" status, consider reviewing your settings for rollups and evaluation windows. For instance, if a monitor has a 4-minute rollup and a 20-minute evaluation window, it produces one data point every 4 minutes, leading to a maximum of 5 datapoints within the window. If the "Require Full Window" option is enabled, the evaluation may result in "No Data" because the window is not fully populated. For most use cases, disable the "Require Full Window" setting unless your specific scenario demands complete data for accurate evaluation. For more information, see [Rollups in monitors](https://docs.datadoghq.com/dashboards/functions/rollup/#rollups-in-monitors). ### Cloud metric delays{% #cloud-metric-delays %} If your monitor queries for crawler-based cloud metrics, use an [evaluation delay](https://docs.datadoghq.com/monitors/configuration/?tabs=thresholdalert#evaluation-delay) to help ensure that the metrics have arrived before the monitor evaluates. Read [cloud metric delay](https://docs.datadoghq.com/integrations/faq/cloud-metric-delay/) for more information about cloud integration crawler schedules. ## Alert conditions{% #alert-conditions %} Unexpected monitor behavior can sometimes be the result of misconfigured [alert conditions](https://docs.datadoghq.com/monitors/configuration/?tabs=thresholdalert#set-alert-conditions), which vary by [monitor type](https://docs.datadoghq.com/monitors/types). If your monitor query uses the `as_count()` function, check the [`as_count()` in Monitor Evaluations](https://docs.datadoghq.com/monitors/guide/as-count-in-monitor-evaluations/) guide. If using recovery thresholds, check the conditions listed in the [recovery thresholds guide](https://docs.datadoghq.com/monitors/guide/recovery-thresholds/#behavior) to see if the behavior is expected. ### New group delays{% #new-group-delays %} If you anticipate creating new monitor groups within the scope of your multi alert monitors, you may want to configure a delay for the evaluation of these new groups. This can help you avoid alerts from the expected behavior of new groups, such as high resource usage associated with the creation of a new container. Read [new group delay](https://docs.datadoghq.com/monitors/configuration/?tabs=thresholdalert#new-group-delay) for more information. ## Monitor groups and retention{% #monitor-groups-and-retention %} By default, Datadog keeps monitor groups available in the UI for 24 hours, or 48 hours for host monitors, unless the query is changed. This retention period affects how long monitor groups remain visible and continue to be evaluated after they stop reporting data. For detailed information about monitor group persistence, including how to handle renamed or decommissioned hosts that continue appearing in alerts, see [Monitor settings changes not taking effect](https://docs.datadoghq.com/monitors/guide/why-did-my-monitor-settings-change-not-take-effect). ## Notification issues{% #notification-issues %} If your monitor is behaving as expected, but producing unwanted notifications, there are multiple options to reduce or suppress notifications: - For monitors that rapidly change between states, read [reduce alert flapping](https://docs.datadoghq.com/monitors/guide/reduce-alert-flapping/) for ways to minimize alert fatigue. - For alerts which are expected or are otherwise not useful for your organization, use [Downtimes](https://docs.datadoghq.com/monitors/guide/suppress-alert-with-downtimes/) to suppress unwanted notifications. - To control alert routing, use [template variables](https://docs.datadoghq.com/monitors/notify/variables/?tab=is_alert&tabs=is_alert#template-variables) and the separation of **warning** or **alert** states with [conditional variables](https://docs.datadoghq.com/monitors/notify/variables/?tab=is_alert&tabs=is_alert#conditional-variables). ### Missing notifications{% #missing-notifications %} If you suspect that notifications are not being properly delivered, check the items below to ensure that notifications are able to be delivered: - Check [email preferences](https://docs.datadoghq.com/account_management/#preferences) for the recipient and ensure that `Notification from monitor alerts` is checked. - Check the [event stream](https://docs.datadoghq.com/events/stream) for events with the string `Error delivering notification`. ### Opsgenie multi-notifications{% #opsgenie-multi-notifications %} If you are using multiple `@opsgenie-[...]` notifications in your monitor, we send those notifications with the same alias to Opsgenie. Due to an [Opsgenie feature](https://docs.opsgenie.com/docs/alert-deduplication), Opsgenie will discard what is seen as a duplication. ## Further Reading{% #further-reading %} - [Alert on no change in value](https://docs.datadoghq.com/monitors/guide/alert-on-no-change-in-value/) - [Set up an alert for when a specific tag stops reporting](https://docs.datadoghq.com/monitors/guide/set-up-an-alert-for-when-a-specific-tag-stops-reporting/) - [Prevent alerts from monitors that were in downtime](https://docs.datadoghq.com/monitors/guide/prevent-alerts-from-monitors-that-were-in-downtime/) - [Enable preconfigured alerts with monitor templates](https://www.datadoghq.com/blog/datadog-recommended-monitors/) - [Monitor alerts and events with OpsGenie and Datadog](https://www.datadoghq.com/blog/datadog-recommended-monitors/) - [Monitoring services and setting SLAs with Datadog](https://www.datadoghq.com/blog/set-and-monitor-slas/)