New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Event Stream

The event stream is a display of the most recent events generated by your infrastructure and the associated monitors.

Event Query Language

Narrow your search by filtering on certain event properties. See the list of filters below for more details. Note: Filters perform an exact match search. Partial strings are not considered.

FilterDescription
sources:github,chefShow events from GitHub OR Chef.
tags:env-prod OR dbShow events tagged with #env-prod OR #db.
tags:security-group:sg-123 AND role:common-nodeShow events tagged with #security-group:sg-123 AND #role:common-node.
hosts:i-0ade23e6,db.myapp.comShow events from i-0ade23e6 OR db.myapp.com.
status:errorShow events with error status. (supports: error, warning, success)
priority:lowShow only low-priority events. (supports: low or normal. defaults to all)
cloud_provider:* NOT "azure"Show all cloud providers except the ones tagged with “azure”

Full text search works on all keywords provided in the search query after applying any filters. Full text search looks inside the event text, title, tags, users who commented on the event, host names, and devices tied to the event.

Use tag search to find all events with the same key tag with the following query:

FilterDescription
tags:<KEY>:<VALUE>Shows events with the <KEY>:<VALUE> tag.
tags:<VALUE>Shows all events with the <VALUE> attached, whatever the <KEY>.
<KEY>:*Shows all events with the <KEY> attached.
<KEY>:<REGEX>Shows all events with <KEY>:<VALUE> tag where <VALUE> matches the <REGEX>
tags:<KEY>Doesn’t return anything.
<KEY>:<VALUE>Doesn’t return anything.

To combine multiple terms into a complex query, use any of the following Boolean operators:

OperatorDescriptionExample
ANDIntersection: both terms are in the selected events (for tags, if nothing is added, AND is taken by default).redis_* AND down
ORUnion: either term is contained in the selected events. Use a comma (,) for tags.sources:nagios,chef directory OR Mixlib
NOTExclusion: the following term is NOT in the event. This operator works for strings only—use - in front of tags.-tags:<KEY>:<VALUE> NOT "<STRING>"

Note: Some of the advanced query language features (e.g. boolean logic) work only in the event stream page, and are not available in graph tiles or dashboard widgets.

Combine prefixes to construct more complex searches. For example, if you wanted to find all open chef or nagios errors that mention cassandra, you’d have a search like:

sources:nagios,chef status:error cassandra

Note: Do not use spaces after the colon or commas in these lists. Anything not attached to a prefix goes to full text search.

Show events unaggregated

To show unaggregated events in your event stream, un-check the Aggregate related events toggle in the upper right corner of your event stream:

Events email

When you need to integrate an application or system with Datadog, you have a few choices. The first is using one of the existing Datadog integrations. This gets you access to a wide variety of metrics and events with minimal configuration effort on your part. If your application isn’t one of the integrated applications, opt to create a custom check using the Agent. This requires more effort and potentially more knowledge on how the application and Datadog work. You have the option to email events to Datadog if your an application does not have an existing Datadog integration, and you don’t want to create an Agent check. There are two different ways to use events with email, depending on whether the application offers you the ability to customize the format of the email body being sent. There is another option available if you aren’t using an application that has an integration, and you don’t want to create an Agent check. Rely on your application or system sending an email instead. There are two different ways to use events via email, depending on whether the application offers you the ability to customize the format of the email body being sent.

JSON-Formatted vs Plain Text:
If you have complete control over the email sent by the application to Datadog, then you probably want to configure a JSON-formatted message to be sent. This allows you to set everything in the event that appears in the event stream. See below for examples of each.

Plain Text Email

Source Email

In the source plain text email, you only have three fields to control: sender email address (required), subject (required), and body (optional).

Sender's email: Matt@Datadog
Subject: Env:Test - System at 50% CPU - #test
Body: This is a test message showing that env:test is at 50% CPU - #test

Datadog Event

The subject of the email becomes the title of the event and the body of the email becomes the body of the event. Although it looks like a tag appears at the end of the title and body of the event, neither instance are actually tags. The sender of the email appears at the bottom of the event.

JSON Email

Source Email

In the source JSON-formatted email, the following fields are available to control:

Note: If your JSON is not properly formatted or the email is sent without a subject, the event won’t appear in your Event Stream.

Datadog Event

In a JSON-formatted email, the subject of the email message is irrelevant. It is replaced by the title in the JSON in the body of the email. All data that appears in the event is defined in JSON in the body of the email. This JSON must be well-formed or the message is ignored. This means it should be formatted properly with commas separating key value pairs and it must be pure JSON. Note: If you are testing the email with a standard email client, the body may be converted to HTML as a convenience to the user. This causes the JSON to no longer be JSON and the email is ignored by Datadog.

Setting Up The Email Address

To set up the email, first log in to your Datadog account. From the Integrations menu, choose APIs, then scroll down to Events API Emails. This section shows you all the emails available for your applications and who created them. Choose the format for your messages from the Format dropdown, then click Create API Email.

Markdown events

Datadog event text supports Markdown. Note: Embedding HTML in markdown is not supported in Datadog.

To use Markdown in the event text, start the text block with %%% \n and end the text block with \n %%%

Example:

{
      "title": "Did you hear the news today?",
      "text": "%%% \n [an example link](http://catchpoint.com/session_id \"Title\") \n %%%",
      "priority": "normal",
      "tags": ["environment:test"],
      "alert_type": "info"
}

Note: If you are embedding a link in a Markdown block, make sure the URL is encoded properly.

Example:

# Not encoded
http://catchpoint.com/session_id:123456
# Encoded
http://catchpoint.com/session_id%3A123456

@ notifications

  • @support-datadog - Creates a Datadog support ticket when posted directly to your event stream (including comments).
  • @all – Sends a notification to all members of your organization.
  • @yourname – Notifies the specific user named ‘yourname’.
  • @test@example.com Sends an email to test@example.com.
  • If you have Slack, Webhooks, Pagerduty, or VictorOps, use:
    • @slack-<SLACK_ACCOUNT>-<CHANNEL_NAME> – posts the event or graph to that channel.
    • @webhook – Alerts or triggers whatever is attached to that webhook. Check out our blogpost on Webhooks!
    • @pagerduty – Sends an alert to Pagerduty. You can also use @pagerduty-acknowledge and @pagerduty-resolve.