Notifications

Overview

Notifications are a key component of any monitor. You want to make sure the right people get notified so the problem can be resolved as soon as possible.

Give your monitor a title. It is often useful to use a succinct explanation of the monitor so a notified team member can quickly understand what is going on.
Enter a message for the monitor. This field allows standard Markdown formatting as well as in-line variables and tag variables. Use conditional variables to modulate the notification text and send them to different contacts with the Datadog’s @-notification syntax. A common use-case for the monitor message is to include a step-by-step way to resolve the problem.
Optionally enable monitor renotification. This option is useful to remind your team that a problem is not solved until the monitor is marked as resolved. If enabled, you can configure an escalation message to be sent anytime the monitor renotifies. The original message is included as well.

Note: To avoid notification storms we now group notifications with the same monitor ID and alert type in 20 second buckets. The first two notifications in the group within a 20 second bucket are sent as normal. All additional notifications within that 20 second window are sent as a single message.

Variables

Use variables to customize your monitor notifications, the available variables are:

Variable	Description
`{{value}}`	Display the value that breached the alert.
`{{threshold}}`	Display the alert threshold selected in the monitor’s Set alert conditions section.
`{{warn_threshold}}`	Display the warning threshold selected in the monitor’s Set alert conditions section if any.
`{{ok_threshold}}`	Display the value that recovered the monitor.
`{{comparator}}`	Display the relational value selected in the monitor’s Set alert conditions section.

Tag variables

For multi-alert monitors, add tag variables to include information in your alert message that’s specific to the tag scope that triggered the alert. This is dependent on the tags you used in the trigger a separate alert for each field in Section 1 of your monitor.

For example, if you trigger by each host tag, then a number of tag variables related to host are available to you in your Section 3: Say what’s happening, such as {{host.name}}, {{host.ip}}, etc..

This also works for custom tags as well. If you’ve added a custom tag that follows the key:value syntax, then you can group data by those tag keys.

This means you are able to apply the tag’s key to the list of tags that a multi-alert has a separate trigger for—which in turn means you can use that tag’s .name variable in your monitor message.

Note:

Template variable content is escaped by default. If your variable contains JSON or code that you would NOT like to be escaped, use triple braces instead of double braces (e.g. {{{event.text}}}).
See a complete list of contextual template variables available to your monitor by clicking the Use message template variables link or in the list of suggestions that appears when you type {{ to begin a template variable name. The variables available are different depending on the combination of metric, tags, and other features of the monitor you are working on.
The tag template variables can also be used in the monitor titles (names), but the variables are only populated in the text of Datadog child events (not the parent, which displays an aggregation summary).
Some tags identifying your triggering scope are automatically inserted into the title of your multi alert.

Examples

Here’s an example of where a user had a number of hosts tagged by different creator: values, e.g, creator:wes_anderson and creator:saint_exupéry.

Here, the user was able to set up a multi-alert monitor to trigger a separate alert for each creator: tag, so they were able to include the {{creator.name}} in their monitor message. When this monitor triggers, the recipient of the alert notification sees whether the monitor was triggered by wes_anderson, saint_exupéry, or some other creator: value.

Here is an example of how you can use template variables for a multi-alert:

and the corresponding event notification:

Tag key with period

If your tag group’s key has a period in it, you have to hardwire your template variables to include brackets around the full key. For example, if you submit a metric tagged with dot.key.test:five and then set up a multi-alert monitor triggered by the dot.ket.test group tag, you have to apply the following syntax in order to use the dot.key.test.name tag variable:

Conditional variables

Conditional variables allow for different text to be sent to different contacts based on the state of the monitor and the details of how it was triggered. These condition variables can be used within either the subject or body of the notification set in section 3 of the monitor definition.

Keep in mind when using conditional tags that they must have an open (example: {{#is_alert}}) and closing (example: {{/is_alert}}) pair with the desired text and @ mentions in between.

The conditional variables available are:

Conditional Variable	Description
`{{is_alert}}`	Show when monitor alerts
`{{^is_alert}}`	Show unless monitor alerts
`{{is_match}}`	Show when the context matches a string
`{{^is_match}}`	Show unless the context matches a string
`{{#is_exact_match}}`	Show when the context matches a string exactly
`{{#is_no_data}}`	Show when monitor notifies on missing data
`{{^is_no_data}}`	Show unless monitor notifies on missing data
`{{#is_warning}}`	Show when monitor warns
`{{^is_warning}}`	Show unless monitor warns
`{{#is_recovery}}`	Show when monitor recovers
`{{^is_recovery}}`	Show unless monitor recovers
`{{#is_warning_recovery}}`	Show when monitor recovers from a warning
`{{^is_warning_recovery}}`	Show unless monitor recovers from a warning
`{{^is_alert_recovery}}`	Show unless monitor recovers from an alert
`{{#is_alert_to_warning}}`	Show when monitor transitions from alert to warning
`{{^is_alert_to_warning}}`	Show unless monitor transitions from alert to warning
`{{#is_no_data_recovery}}`	Show when monitor recovers from a no data
`{{^is_no_data_recovery}}`	Show unless monitor recovers from a no data

These can also be seen in the “Use message template variables” help box in Step 3 of the monitor editor.

Here are some examples of their usage:

These variables use simple if-else logic to display a different message depending on the event type (warning, recovery, no data…)

Here is an example of how you can set it up in the editor:

The corresponding trigger event notification looks like this:

and the recovery notification:

{{is_recovery}} triggers when a monitor recovers indifferently either from a WARNING state or an ALERT state.
{{is_alert_recovery}} triggers when a monitor recovers directly from an ALERT state to an OK state.
{{is_warning_recovery}} triggers when a monitor recovers from a WARNING state to an OK state

This means that if the monitor switches from an ALERT to a WARNING to an OK state:

the {{is_recovery}} would trigger
the {{is_alert_recovery}} wouldn’t trigger
the {{is_warning_recovery}} would trigger.

The {{is_match}} conditional allows you to match the triggering context to any given string in order to display a different message in your notifications. Use any of the available tag variables in your conditional statement. A match is made if the comparison string is anywhere in the resolved variable.

Tag variables use the following format:

{{#is_match "<TAG_VARIABLE>.name" "<COMPARISON_STRING>"}}
  This shows if <COMPARISON_STRING> is included in <TAG_VARIABLE>
{{/is_match}}

For example, if you want to notify your DB team if a triggering host has the role:db_cassandra tag or the role:db_postgres tag, use the following:

{{#is_match "role.name" "db"}}
  This shows only if the host that triggered the alert has `role` tag variable with `db` in it. 
  It would trigger for role:db_cassandra and role:db_postgres
{{/is_match}}

Note: To check if a <TAG_VARIABLE> is NOT empty, use the {{is_match}} conditional with an empty string.

{{#is_match "<TAG_VARIABLE>.name" ""}}
  This shows if <TAG_VARIABLE> is not empty.
{{/is_match}}

The {{is_exact_match}} conditional looks for the exact string in the tag variable, rather than using substring matching. The variable uses the following format:

{{#is_exact_match "<TAG_VARIABLE>.name" "<COMPARISON_STRING>"}}
  This shows if <COMPARISON_STRING> is exactly <TAG_VARIABLE>.
{{/is_exact_match}}

For instance, if an alert that can be triggered by two hosts tagged with role:production and role:production-1:

  {{#is_match "role.name" "production"}}
    This shows only if the host that triggered the alert has the tags role:production or the role:production attached.
  {{/is_match}}

  {{#is_exact_match "host.name" "production"}}
    This shows only if the host that triggered has the tag role:production attached.
  {{/is_exact_match}}

@-notification

Send the monitor notification to the appropriate endpoint:

Notify a Datadog user via email by adding @<DD_USER_EMAIL> in your notification message.
Notify any non-Datadog users via email by adding @<EMAIL> to the notification message.
Install Slack or Hipchat integration to send your notifications directly in the appropriate channel.

After having installed the Slack integration, type @slack in your notification message to see the available list of channels to send your notification to.

@-mentions in Slack from monitor alert:

Wrap the @username in < > as seen below in your monitors message template to @ notify the defined user within slack notifications. For example this configuration:

Would produce this slack message:

You can also mention @here or @channel using <!here> or <!channel>, respectively.

For user groups, use <!subteam^GROUP_ID|GROUP_NAME>. To find the GROUP_ID, query the usergroups.list API endpoint of Slack. For example, for a user group named testers you would use the following syntax:

<!subteam^12345|testers>

Note: Trailing special characters in a channel name are unsupported for the Slack @-notifications. e.g. @----critical_alerts works, but @--critical_alerts-- won’t receive any notifications.

Using message template variables to dynamically create @-mentions:

Use message template variables within a monitor message to dynamically build @-mentions.

For example, if the rendered variable is setup as a channel in the Slack integration:

@slack-{{owner.name}} post a slack DM message directly the owner of this monitor.
@slack-{{host.name}} post a slack message to the #host.name channel in Slack.

After having installed the Hipchat integration, type @hipchat in your notification message to see the available list of channels to send your notification to.

Use @here in the monitor message to notify everybody in a given Hipchat channel. Do not use @all as it notifies all members of your Datadog organization.

Note: Hipchat users must have room notifications set to at least Normal to receive notifications this way.

Advanced notification configuration

Include links to appropriate dashboards

Many organizations like to include additional context to their Alerts. Quick links to relevant dashboards as a part of the Alert have proven to reduce the overall time it takes during the break fix process to reduce time to resolution.

Datadog makes message template variables available to each defined monitor. Using these variables, you can dynamically build a URL that links Datadog users to an appropriate dashboard using the scope of the monitor.

Here are a few examples of providing links to items like System Dashboards, Integration Dashboards, HostMaps, and Managed Monitors pages.

The first example to review is the most common. Let’s say you would like to provide a link to a System Dashboard when a monitor for a specific system metric has exceeded your defined threshold. The message template variable that can be leveraged in this instance is {{host.name}}. Include the following URL as a part of your Monitor “Say What’s Happening” section:

https://app.datadoghq.com/dash/integration/system_overview?tpl_var_scope=host:{{host.name}}

As you can see, {{host.name}} is replaced with the offending host of the monitor in question.

Find below additional examples of links that could be added to Monitors to provide Datadog users quick access to common pages leveraged during the break, fix, and triage process:

To include a link to the HostMap to compare metrics with other similar hosts, use a link like below to be included in your Monitor:

https://app.datadoghq.com/infrastructure/map?fillby=avg%3Acpuutilization&sizeby=avg%3Anometric&filter=cassandra

The above link has more customizable options than your standard System Dashboard. Here you have additional variables to define. Most common variables passed into this URL are the following fillby, sizeby, filter:

fillby is defined by adding fillby:avg:<MetricName>.
sizeby is defined by adding sizeby:avg:<SecondMetricName>.
filter is used to specify a specific integration (i.e. Cassandra, mysql, apache, snmp, etc) by adding filter=<integration_name>.

In the example below, colors fill the Hostmap hexagons by system.cpu.system. The hexagons are sized by system.cpu.stolen, and they are filtered to only include Cassandra hosts.

To link to a “Manage Monitors” page that displays all of the monitors for the host in question, define a link like below:

https://app.datadoghq.com/monitors/manage?q=scope:host:{{host.name}}

The above URL links to all monitors for this host. You have other options available to further refine the link.

For example, if you would only like monitors that are in an Alert State, you can add the following status:Alert (other statuses that can be leveraged are WARN, NO%20DATA, OK and MUTED). Below is an example link:

https://app.datadoghq.com/monitors/manage?q=scope:host:{{host.name}}&status:Alert

If you would like all monitors for a specific application or integration, add the following query to the URL q=<integration_name>:

https://app.datadoghq.com/monitors/manage?q=cassandra

If you are building application- or integration-specific monitors, link to that specific Integration Dashboard as well as adding a scope for the host that triggered the monitor.

In the example below, all that is necessary to populate is the <integration_name> section for something like Cassandra, Apache, SNMP, etc., as well as providing the scope for the offending host:

https://app.datadoghq.com/dash/integration/<integration_name>?tpl_var_scope=host:{{host.name}}