Metric monitors are one of the most commonly used type of monitor. This guide clarifies the change alert detection method’s behavior and its additional options. Learn how change alert monitors work and how to troubleshoot change alert evaluations.
What are change alert monitors?
Here is a breakdown of how monitors with the change detection method work:
- The monitor takes a query of data points at the current time.
- It takes a query of data points N minutes, hours, or days ago.
- Then, it takes a query of the difference of the values between (1) and (2).
- Aggregation is applied over the query in (3) which returns a single value.
- The threshold defined in Set alert conditions is compared to the single value returned in (4).
Here are the different options that you need to configure in a change alert monitor.
The example shows the following alert condition:
The average of the change over 1 hour compared to 5 minutes
|average||The aggregation that is used on the query.|
|change||Choose between the absolute or percentage change of the value.|
|1 hour||The evaluation window. For more information, see the Monitor Configuration documentation.||This can be N minutes, hours, days, weeks, or at most one month.|
|5 minutes||The timeframe that you wish to shift the query by.||This can be N minutes, hours, days, weeks, or at most one month ago.|
Change and change %
There are two options when configuring a change alert detection, Change and % Change.
This determines the way the monitor evaluates as expressed in the formula section in the following table:
|Change||The absolute change of the value.|
a - b
|% Change||The percentage change of the value compared to its previous value.|
((a - b) / b) * 100
In both cases,
% Change can be either positive or negative.
Troubleshooting a change alert evaluation
To verify the results of your change alert evaluation, reconstruct the metric queries with a Notebook.
Take this change alert monitor with the following settings.
pct_change(avg(last_5m),last_30m):<METRIC> > -50
This is a break down of the query with the following conditions:
- Aggregation of avg.
- Uses % change.
- Evaluation window of 5 minutes.
- Timeshift of 30 minutes or 1800 seconds.
- Threshold of > -50.
Reconstructing the query
- Use a notebook and the timeshift function to reconstruct the data used by the monitor at a specific evaluation.
- Query of data points at the current time (this is the normal query ).
- Query of data points N minutes ago (this is the normal query + timeshift(-1800)).
- The timeshift function uses a negative duration because you’re shifting the data back. Combine these queries along with the % change formula from the table.
- Note: Since this example only has one metric, it’s also possible to use a single query (a) and add the formula
((a - timeshift(a, -1800)) / timeshift(a, -1800)) * 100
- Compare the monitor’s history graph with the notebook graph. Are the values comparable?
- Apply the aggregation.
- To compare your notebook graph to the change alert monitor evaluation, scope your timeframe to match the change alert.
- For example, if you are looking to verify the value of a monitor evaluation over the last five minutes at 1:30, scope your notebook to 1:25 - 1:30.