Live Search
Security Monitoring is now available Security Monitoring is now available

Live Search

Live Search

Overview

The APM Live Search gives you the ability to search all ingested spans using any tag in real-time for the past 15 minutes. It displays spans as soon as they are sent by the Datadog agent and before they are indexed by Datadog. All spans ingested by Datadog are displayed without any sampling (Tracing without Limits™). With the APM Live Search you can:

  • Write search queries to refine the stream of traces by any tag on any span. For example, monitor that a new deployment went smoothly by filtering on version_id of all tags
  • View 100% of trace spans as they are ingested. For example, view outage related information in real-time as it happens related to specific org_id or customer_id associated with a child span. Note that there is no sampling on ingested spans for 15 minutes.
  • Search queries autocomplete in real-time. For example, check if a process has correctly started type process_id of a child span tag and it auto completes the ID.
  • View Live timeseries visualization of key RED metrics: requests per second, errors, and latency. For example, monitor load test and performance impact on your endpoints by filtering on the duration of a child resource.
  • One-click search queries on any span or tag directly from the trace panel view.
  • Add, remove, and sort columns from span tags for a customized view.

Live Search Mode

Live Search mode

Live Search mode is the default experience on the Traces page or you can choose the LIVE option in the time range selector to switch to the Live Search mode for the past 15 minutes from the HISTORICAL mode. The number of received spans per second is displayed at the top of the traces table, as well as the sampling rate. Since a stream of thousands of spans per second is not human readable, high throughput span streams are sampled for visual clarity but are still searchable. Use the Live Search query bar filtering features to filter the spans stream and the Pause/Play button at the top right of the screen to pause or resume the stream.

Note: Selecting any span pauses the stream and displays more details about the selected span in the trace side panel.

Filtering the Trace Stream and Search Query

Live Search query

A valid query in the search bar displays traces that match your search criteria across all spans. The search syntax is the same in the Live Search views as in the other trace views, but here, your query is matched against all of the ingested traces across any span and any tag, and not just the indexed ones.

Note: You can select only the top-level spans of the service by selecting the checkbox above the trace table. You can use this feature on high traffic applications to reduce the number of spans displayed and view only the entry point spans of the services. Selecting this box only visually filters the spans shown.

You can also filter on attributes that are not defined as facets. For example, to filter on the customer.id attribute, there are two options:

  • Click on the attribute in the trace panel and add it to the search query @customer.id:584959.

  • Filter on all spans with a customer.id attribute by typing “customer.id” in the search query bar: @customer.id:584959

Further Reading