Notifications are a key component of any monitor. You want to make sure the right people get notified so the problem can be resolved as soon as possible.
Give your monitor a title. It is often useful to use a succinct explanation of the monitor so a notified team member can quickly understand what is going on.
Enter a message for the monitor. This field allows standard Markdown formatting as well as in-line variables and tag variables. Use conditional variables to modulate the notification text and send them to different contacts with the Datadog’s @-notification syntax. A common use-case for the monitor message is to include a step-by-step way to resolve the problem.
Optionally enable monitor renotification. This option is useful to remind your team that a problem is not solved until the monitor is marked as resolved. If enabled, an escalation message can be configured to send any time the monitor renotifies. The original message is included as well.
Note: To avoid notification storms, Datadog groups notifications with the same monitor ID and alert type in 20 second buckets. The first two notifications in the group within a 20 second bucket are sent as normal. All additional notifications within that 20 second window are sent as a single message.
Use variables to customize your monitor notifications, the available variables are:
| Variable | Description |
|---|---|
{{value}} |
Display the value that breached the alert. |
{{threshold}} |
Display the alert threshold selected in the monitor’s Set alert conditions section. |
{{warn_threshold}} |
Display the warning threshold selected in the monitor’s Set alert conditions section if any. |
{{ok_threshold}} |
Display the value that recovered the monitor. |
{{comparator}} |
Display the relational value selected in the monitor’s Set alert conditions section. |
{{last_triggered_at}} |
Display the UTC date/time when the monitor last triggered. |
Note: When entering decimal values for thresholds, if your value is <1, add a leading 0 to the number. For example, use 0.5, not .5.
For multi-alert monitors, add tag variables to include information in your alert message that’s specific to the tag scope that triggered the alert. This is dependent on the tags you used in the trigger a separate alert for each field in Section 1 of your monitor.
For example, if you trigger by each host tag, then a number of tag variables related to host are available to you in Section 3: Say what’s happening, such as {{host.name}}, {{host.ip}}, etc..
This also works for custom tags. If a custom tag is added that follows the key:value syntax, then data can be grouped by those tag keys. This enables a multi-alert (separate trigger) for each value. Additionally, the tag’s .name variable can be used in your monitor message.
Note:
Template variable content is escaped by default. If your variable contains JSON or code that you would NOT like to be escaped, use triple braces instead of double braces (e.g. {{{event.text}}}).
See a complete list of contextual template variables available to your monitor by clicking the Use message template variables link or in the list of suggestions that appears when you type {{ to begin a template variable name. The variables available are different depending on the combination of metric, tags, and other features of the monitor you are working on.
The tag template variables can also be used in the monitor titles (names), but the variables are only populated in the text of Datadog child events (not the parent, which displays an aggregation summary).
Some tags identifying your triggering scope are automatically inserted into the title of your multi alert.
Here’s an example of where a user had a number of hosts tagged by different creator: values, e.g, creator:wes_anderson and creator:saint_exupéry.
Here, the user was able to set up a multi-alert monitor to trigger a separate alert for each creator: tag, so they were able to include the {{creator.name}} in their monitor message. When this monitor triggers, the recipient of the alert notification sees whether the monitor was triggered by wes_anderson, saint_exupéry, or some other creator: value.
This is an example of using template variables for a multi-alert:
and the corresponding event notification:
If your tag group’s key has a period in it, you have to hardwire your template variables to include brackets around the full key.
For example, if you submit a metric tagged with dot.key.test:five and then set up a multi-alert monitor triggered by the dot.ket.test group tag, you have to apply the following syntax in order to use the dot.key.test.name tag variable:
Conditional variables allow for different text to be sent to different contacts based on the state of the monitor and the details of how it was triggered. These condition variables can be used within either the subject or body of the notification set in section 3 of the monitor definition.
Keep in mind when using conditional tags that they must have an open (example: {{#is_alert}}) and closing (example: {{/is_alert}}) pair with the desired text and @ mentions in between.
The conditional variables available are:
| Conditional Variable | Description |
|---|---|
{{#is_alert}} |
Show when monitor alerts |
{{^is_alert}} |
Show unless monitor alerts |
{{#is_match}} |
Show when the context matches a string |
{{^is_match}} |
Show unless the context matches a string |
{{#is_exact_match}} |
Show when the context matches a string exactly |
{{^is_exact_match}} |
Show unless the context matches a string exactly |
{{#is_no_data}} |
Show when monitor notifies on missing data |
{{^is_no_data}} |
Show unless monitor notifies on missing data |
{{#is_warning}} |
Show when monitor warns |
{{^is_warning}} |
Show unless monitor warns |
{{#is_recovery}} |
Show when monitor recovers from either WARNING or ALERT |
{{^is_recovery}} |
Show unless monitor recovers from either WARNING or ALERT |
{{#is_warning_recovery}} |
Show when monitor recovers from a warning to OK |
{{^is_warning_recovery}} |
Show unless monitor recovers from a warning to OK |
{{#is_alert_recovery}} |
Show when monitor recovers from an alert to OK |
{{^is_alert_recovery}} |
Show unless monitor recovers from an alert to OK |
{{#is_alert_to_warning}} |
Show when monitor transitions from alert to warning |
{{^is_alert_to_warning}} |
Show unless monitor transitions from alert to warning |
{{#is_no_data_recovery}} |
Show when monitor recovers from a no data |
{{^is_no_data_recovery}} |
Show unless monitor recovers from a no data |
These can also be seen in the “Use message template variables” help box in Step 3 of the monitor editor.
Here are some examples of their usage:
These variables use simple if-else logic to display a different message depending on the event type (warning, recovery, no data…)
This is an example in the editor:
The corresponding trigger event notification looks like this:
and the recovery notification:
{{#is_recovery}} triggers when a monitor recovers indifferently either from a WARNING state or an ALERT state.{{#is_alert_recovery}} triggers when a monitor recovers directly from an ALERT state to an OK state.{{#is_warning_recovery}} triggers when a monitor recovers from a WARNING state to an OK stateThis means that if the monitor switches from an ALERT to a WARNING to an OK state:
{{#is_recovery}} would trigger{{#is_alert_recovery}} wouldn’t trigger{{#is_warning_recovery}} would trigger.The {{is_match}} conditional allows you to match the triggering context to any given string in order to display a different message in your notifications.
Use any of the available tag variables in your conditional statement. A match is made if the comparison string is anywhere in the resolved variable.
Tag variables use the following format:
{{#is_match "<TAG_VARIABLE>.name" "<COMPARISON_STRING>"}}
This shows if <COMPARISON_STRING> is included in <TAG_VARIABLE>
{{/is_match}}
For example, if you want to notify your DB team if a triggering host has the role:db_cassandra tag or the role:db_postgres tag, use the following:
{{#is_match "role.name" "db"}}
This shows only if the host that triggered the alert has `role` tag variable with `db` in it.
It would trigger for role:db_cassandra and role:db_postgres
{{/is_match}}
Note: To check if a <TAG_VARIABLE> is NOT empty, use the {{is_match}} conditional with an empty string.
{{#is_match "<TAG_VARIABLE>.name" ""}}
This shows if <TAG_VARIABLE> is not empty.
{{/is_match}}
The {{is_exact_match}} conditional looks for the exact string in the tag variable, rather than using substring matching. The variable uses the following format:
{{#is_exact_match "<TAG_VARIABLE>.name" "<COMPARISON_STRING>"}}
This shows if <COMPARISON_STRING> is exactly <TAG_VARIABLE>.
{{/is_exact_match}}
For instance, if an alert that can be triggered by two hosts tagged with role:production and role:production-1:
{{#is_match "role.name" "production"}}
This shows only if the host that triggered the alert has the tags role:production or the role:production attached.
{{/is_match}}
{{#is_exact_match "host.name" "production"}}
This shows only if the host that triggered has the tag role:production attached.
{{/is_exact_match}}
If your alert message needs to send double curly braces, such as {{ <TEXT> }}, use the {{{{raw}}}} formatting:
The following template:
{{{{raw}}}}
{{ <TEXT_1> }} {{ <TEXT_2> }}
{{{{/raw}}}}
outputs:
{{ <TEXT_1> }} {{ <TEXT_2> }}
The ^|# helpers shown in the Conditional variables section cannot be used with {{{{raw}}}} formatting and must be removed. For instance, to output raw text with the {{is_match}} conditional variable use the following template:
{{{{is_match "host.name" "<HOST_NAME>"}}}}
{{ .matched }} the host name
{{{{/is_match}}}}
If host.name matches <HOST_NAME>, the template outputs:
{{ .matched }} the host name
Send the monitor notification to the appropriate endpoint:
@<DD_USER_EMAIL> in your notification message.@<EMAIL> to the notification message.Note: A @-mention must have a space between it and the last line character: {{value}}@slack-channel is invalid {{value}} @slack-channel is valid.
After having installed the Slack integration, type @slack in your notification message to see the available list of channels to send your notification to.
@-mentions in Slack from monitor alert:
Wrap the @username in < > as seen below in your monitors message template to @ notify the defined user within slack notifications.
For example this configuration:
Would produce this slack message:
Mention @here or @channel by using <!here> or <!channel>, respectively.
For user groups, use <!subteam^GROUP_ID|GROUP_NAME>. To find the GROUP_ID, query the usergroups.list API endpoint of Slack. For example, for a user group named testers you would use the following syntax:
<!subteam^12345|testers>
Note: Trailing special characters in a channel name are unsupported for the Slack @-notifications.
e.g. @----critical_alerts works, but @--critical_alerts-- won’t receive any notifications.
Using message template variables to dynamically create @-mentions:
Use message template variables within a monitor message to dynamically build @-mentions.
For example, if the rendered variable is setup as a channel in the Slack integration:
@slack-{{owner.name}} post a message on the owner’s channel for this monitor.
@slack-{{host.name}} post a slack message to the #host.name channel in Slack.
After you define your monitor, you can test what your monitor’s notification would look like in any applicable state. You can test what’s specified in the “Say What’s Happening” field by selecting “Select all” or selecting specific states.
You can only test states that are available in the monitor’s configuration and test for thresholds specified in the alerting conditions. Recovery thresholds are an exception, as Datadog sends a recovery notification once the monitor is no longer in alert or has no warn conditions.
In the following, you are only able to test for ALERT because only an alert threshold has been specified:
In this next example, you can test for both ALERT and NO DATA:
Message variables auto-populate with an available randomly selected group based on the scope of your monitor’s definition.
Test notifications produce events that can be searched for within the event stream. These notifications also indicate who initiated the test in the message body, and [TEST] is added into the test notification title.
Your test notifications are sent to any notification handle available in the message box. You can also test webhooks.
TIP: Datadog recommends that you test using a test notification channel if you do not want to add these notifications to your on-call channels.
You do not need to save the monitor to run the notification test. Clicking Run test sends the notification, and you can continue to edit the monitor and notification box before saving.
Testing notifications are currently supported for the following monitor types: Host, Metric, Anomaly, Outlier, Forecast, Integration (check only), Process (check only), Network (check only), Custom check, and Event.
Many organizations like to include additional context to their Alerts. Quick links to relevant dashboards as a part of the Alert have proven to reduce the overall time it takes during the break fix process to reduce time to resolution.
Datadog makes message template variables available to each defined monitor. These variables enable dynamic URL building that links Datadog users to an appropriate dashboard using the scope of the monitor.
Here are a few examples of providing links to items like System Dashboards, Integration Dashboards, Host Maps, and Managed Monitors pages.
Example: provide a link to a System Dashboard when a monitor for a specific system metric has exceeded a defined threshold. The message template variable that can be leveraged in this instance is {{host.name}}. Include the following URL as a part of your Monitor “Say What’s Happening” section:
https://app.datadoghq.com/dash/integration/system_overview?tpl_var_scope=host:{{host.name}}
{{host.name}} is replaced with the offending host of the monitor in question.
Find below additional examples of links that could be added to Monitors to provide Datadog users quick access to common pages leveraged during the break, fix, and triage process:
To include a link to the HostMap to compare metrics with other similar hosts, use a link like below to be included in your Monitor:
https://app.datadoghq.com/infrastructure/map?fillby=avg%3Acpuutilization&sizeby=avg%3Anometric&filter=cassandra
The above link has more customizable options than your standard System Dashboard. Here you have additional variables to define. Most common variables passed into this URL are the following fillby, sizeby, filter:
fillby is defined by adding fillby:avg:<MetricName>.sizeby is defined by adding sizeby:avg:<SecondMetricName>.filter is used to specify a specific integration (i.e. Cassandra, mysql, apache, snmp, etc) by adding filter=<integration_name>.In the example below, colors fill the Hostmap hexagons by system.cpu.system. The hexagons are sized by system.cpu.stolen, and they are filtered to only include Cassandra hosts.
To link to a “Manage Monitors” page that displays all of the monitors for the host in question, define a link like below:
https://app.datadoghq.com/monitors/manage?q=scope:host:{{host.name}}
The above URL links to all monitors for this host. You have other options available to further refine the link.
For example, to see all monitors in an Alert State, add status:Alert (other statuses are WARN, NO%20DATA, OK and MUTED). Below is an example link:
https://app.datadoghq.com/monitors/manage?q=scope:host:{{host.name}}&status:Alert
If you would like all monitors for a specific application or integration, add the following query to the URL q=<integration_name>:
https://app.datadoghq.com/monitors/manage?q=cassandra
If you are building application- or integration-specific monitors, link to that specific Integration Dashboard as well as adding a scope for the host that triggered the monitor.
In the example below, all that is necessary to populate is the <integration_name> section for something like Cassandra, Apache, SNMP, etc., as well as providing the scope for the offending host:
https://app.datadoghq.com/dash/integration/<integration_name>?tpl_var_scope=host:{{host.name}}
Additional helpful documentation, links, and articles:
