Migrating to the New Events Features

Datadog's legacy event stream and event monitors retire on June 30, 2022. Datadog is migrating all customers to a new and improved events experience. This page contains important information about this migration. Before the retirement date, follow the steps on this page to ensure that your existing event visualizations and monitors continue to work properly.

Why change Events features?

Launched more than 10 years ago, Datadog’s event stream is one of its earliest features. The new events experience includes many new features that allow you to get even more value from your events. These include event analytics, the ability to generate metrics from your events, the ability to create pipelines to post-process events, and a much friendlier and more intuitive query syntax that is better aligned with other Datadog products, such as Log Management and APM.

What is the migration timeline?

March 2022 - New Events Explorer and analytics are accessible. Datadog begins to migrate customer dashboards and monitors that are not managed with APIs.

May 5, 2022 - The event stream is retired in favor of the Event Explorer.

May 19, 2022 - Starting on this date, while Datadog continues to evaluate event monitors that have not been migrated, they are no longer editable. New event monitors must use the new syntax.

June 30, 2022 - Datadog stops evaluating event monitors that have not been migrated. Legacy event monitors stop working.

What action do I need to take?

If you do not manage your dashboard or monitors using external API-based tools (such as Terraform or scripts), then no action is required on your end. Datadog will migrate your dashboards and monitors before April 30, 2022. Datadog will leave your old monitors in place but they will be muted and Datadog will stop evaluating them by June 30, 2022 at the latest.

If you use Terraform or other API-based scripts to manage all or some of your dashboards, Datadog will migrate queries in your event widgets and overlays to the new syntax, but you will need to update your scripts to keep them in sync before June 30, 2022.

If you use Terraform or other API-based scripts to manage all or some of your monitors, you have until June 30, 2022, to update them. After this date, Datadog will create new versions of any non-migrated monitors and mute the existing monitors to ensure that you continue to have alerting.

Datadog can also assist you with migrating your monitors by suggesting updates or by applying the updates to your monitors.

What’s new?

Events Explorer

The Events Explorer displays the most recent events generated by your infrastructure and services or monitoring alerts. It replaces the Events Stream and provides a friendlier and more intuitive query syntax. See Events Explorer to learn more.

Event Analytics

Events Analytics display filtered by 'source:cloudtrail'

In addition to viewing and searching events in the Explorer, you can now graph as a timeseries, top list or table and group the number of events for a given query. See Event Analytics to learn more.

You may also generate metrics with 15-month retention from any event search query to create monitors and alerts based on historical events.

Image of metrics with the events search query.

Graph events in dashboards

Events Analytics

You can now graph events for a given query inside your dashboards as a timeseries graph, query value, top list, table and more.

For example, check out the Monitor Notifications Overview dashboard, which analyzes monitor alert event trends to help you improve your configuration and reduce alert fatigue.

New event monitors experience

Event monitors have been ported to use the same standardized set of features as other products (Logs, RUM, APM) and have additional capabilities.

When you create event monitors, the new query search field uses autocomplete, rather than the legacy fill-in-the-blank query.

New UI for monitor query syntax

The new query search allows you to use complex queries in event monitors with new capabilities such as Boolean operators or wildcards.

Pipelines

Datadog automatically parses JSON-formatted events. When events are not JSON-formatted, they are parsed and enriched by chaining them sequentially through a processing pipeline. Processors extract meaningful information or attributes from semi-structured text to reuse as facets. Each event that comes through the pipelines is tested against every pipeline filter. If it matches a filter, then all the processors are applied sequentially before moving to the next pipeline.

Reserved attributes

This list describes automatically ingested reserved attributes with events.

AttributeDescription
hostThe name of the originating host as defined in metrics. Datadog automatically retrieves corresponding host tags from the matching host in Datadog and applies them to your events. The Agent sets this value automatically.
sourceThis corresponds to the integration name, or the technology from which the event originated. When it matches an integration name, Datadog automatically installs the corresponding parsers and facets. For example: nginx, postgresql, and more.
statusThis corresponds to the level or severity of an event.
serviceThe name of the application or service generating the events.
messageBy default, Datadog ingests the value of the message attribute as the body of the event entry.

What Changed?

Note: The process for sending events remains the same. You can continue sending events using the API, the Agent, or the events via email feature as before.

Event aggregations are no longer performed or displayed in the UI

Datadog no longer automatically performs Event aggregation, and no longer groups Events by the aggregation_key attribute. The UI no longer displays Event aggregation.

Event comments are no longer supported or displayed in the UI

Comments created using the API with the user_update event type will be displayed as normal events.

Status remapping in queries

Some status values have changed:

Legacy statusNew status
successok
warningwarn
infoinfo
errorerror

Source remapping in queries

Many Event source names have changed. See the full list of affected source names.

The monitor evaluation window is limited to 48 hours.

Your monitors are not evaluated beyond a 48 hour window. If you need to use a longer evaluation window, you can generate custom metrics from events and use a metrics monitor, where the evaluation window can be up to one month.

You can group by only up to 4 facets.

(Previously: Unlimited groups) Top-values, the highest frequency values of a group, are limited based on the total number of groups. For example, if a monitor triggers more times than the facet limit, it sorts by top group and shows only the top N groups. For example, N = 30 resulting hosts if grouping by two facets and one facet is host.

  • One facet results in a limit of 1000 top values.
  • Two facets results in a limit of 30 top values per facet (at most 900 groups)
  • Three facets results in a limit of 10 top values per facet (at most 1000 groups)
  • Four facets results in a limit of 5 top values per group (at most 625 groups)

Recovery thresholds in monitors are no longer supported

Event monitor thresholds no longer support the warning_recovery and critical_recovery threshold types. Recovery thresholds need to be removed in new Event monitors.

If you are using these features, contact Support to get help finding an alternative solution.

Examples

Event query syntax examples before and after

Show events from GitHub or Chef
Legacy syntax
sources:github,chef
New syntax
source:(github OR chef)
Show events tagged with env-prod
Legacy syntax
tags:env-prod
New syntax
tags:env-prod
Show events tagged with #env-prod or #db
Legacy syntax
tags:env-prod,db, tags:env-prod OR db
New syntax
tags:(env-prod OR db)
Show events tagged with #security-group:sg-123 and #role:common-node
Legacy syntax
tags:security-group:sg-123 AND role:common-node
New syntax
tags:(security-group:sg-123 AND role:common-node)
Use wildcards to search for prefixes and suffixes
Legacy syntax
Not available
New syntax
*web matches all event messages ending with web
source:amazon* matches all events that source starts with amazon

Event monitors API syntax examples before and after

The Event monitors API has a new monitor query syntax (See “Event V2 Alert Query” section), with Average and Cardinality rollup methods and fewer required attributes.

No Slack events in the past 24 hours
Legacy syntax
events('priority:all sources:slack').rollup('count').last('1d') < 1
New syntax
events("source:slack").rollup("count").last("1d") < 1
EC2 Instance marked for maintenance
Legacy syntax
events('priority:all "Upcoming AWS maintenance event"').by('name,host').rollup('count').last('2d') >= 1
New syntax
events("Upcoming AWS maintenance event").rollup("count").by("name,host").last("2d") >= 1
Zabbix or Prometheus has triggered an alert for a service today
Legacy syntax
events('tags:service priority:all status:error sources:prometheus sources:zabbix).rollup('count').last('1d') > 0
New syntax
events("source:(prometheus OR zabbix) status:error tags:service").rollup("count").last("1d") > 0
No events received in a datacenter for service datadog-agent
Legacy syntax
Legacy Event Monitors do not support cardinality rollup.
New syntax
events("service:datadog-agent").rollup("cardinality", "datacenter").by("service").last("15m") < 1

Further reading

Additional helpful documentation, links, and articles: