Trace Search
Security Monitoring is now available Security Monitoring is now available

Trace Search

All search parameters are contained in the url of the page, so it is very simple to share your view.

Search syntax

A query is composed of terms and operators.

There are two types of terms:

To combine multiple terms into a complex query, use any of the following boolean operators:

ANDIntersection: both terms are in the selected events (if nothing is added, AND is taken by default)authentication AND failure
ORUnion: either terms is contained in the selected eventsauthentication OR password
-Exclusion: the following term is NOT in the eventauthentication AND -password

To search on a specific facet you must add it as a facet first then add @ to specify you are searching on a facet.

For instance, if your facet name is url and you want to filter on the url value just enter:

Your traces inherit tags from hosts and integrations that generate them. They can be used in the search and as facets as well:

("env:prod" OR test)All traces with the tag #env:prod or the tag #test
(service:srvA OR service:srvB) or (service:(srvA OR srvB))All traces that contain tags #service:srvA or #service:srvB.
("env:prod" AND -"version:beta")All traces that contain #env:prod and that do not contain #version:beta

If your tags don’t follow tags best practices and don’t use the key:value syntax, use this search query:

  • tags:<MY_TAG>


To perform a multi-character wildcard search, use the * symbol as follows:

  • service:web* matches every trace that has a services starting with web
  • @url:data* matches every trace that has a url starting by data.

Numerical values

Use <,>, <=, or >= to perform a search on numerical attributes. For instance, retrieve all traces that have a response time over 100ms with:


It is also possible to search for numerical attributes within a specific range. For instance, retrieve all your 4xx errors with:

@http.status_code:[400 TO 499]


Typing a complex query can be cumbersome. Use the search bar’s autocomplete feature to complete your query using existing values:

Escaping of special characters

The following attributes are considered as special: ?, >, <, :, =,", ~, /, and \ require escaping. For instance, to search traces that contain user=JaneDoe in their url the following search must be entered:


The same logic must be applied to spaces within trace attributes. It is not recommended to have spaces in trace attributes but in such cases, spaces require escaping. If an attribute is called user.first name, perform a search on this attribute by escaping the space:

@user.first\ name:myvalue

Saved Searches

Don’t lose time building the same views everyday. Saved searches contain your search query, columns, and time horizon. They are then available in the search bar thanks to the auto-complete matching whether the search name or query.

To delete a saved search, click on the bin icon under the Trace search drop-down.

Time Range

The time range allows you to display traces within a given time period. Quickly change the time range by selecting a preset range from the dropdown (or entering a custom time frame):

Trace Stream

The Trace Stream is the list of traces that match the selected context. A context is defined by a search bar filter and a time range.

Traces vs Analyzed Spans

In the Trace Stream, select View in App Analytics to view Traces and Analyzed Spans. Choose to display a sampled trace associated with your Analyzed Spans by clicking the Traces button in the upper right corner:

If Traces is selected, Analyzed Spans listed in the trace stream have a sampled trace associated with them. If Analyzed Spans is selected, only the Analyzed Spans are listed in the trace stream.

When a request hits a service (e.g. webserver, database), the Datadog Agent creates an Analyzed Span. It’s a record of the request including its duration, response code, and any custom metadata. An Analyzed Span is represented by a single span with attached metadata for the handled request. For each service that receives a request, the Agent creates an Analyzed Span. If a request runs through a web service, listing service, and database service, the request generates 3 Analyzed Spans. To reduce the amount of Analyzed Spans generated, explicitly turn on/off any Analyzed Span collection for a specific service. To start collecting Analyzed Spans, enable App Analytics for your services.

Displaying a full Trace

Click on any trace to see more details about it:


To add more Trace details to the list, click the Options button and select any Facets you want to see:

Origin resource is a default column that shows the resource at the root of the given trace. To add origin service or origin operation name, click the Options button and select @trace.origin.operation_name or @trace.origin.service.

Multi-line display

Choose to display one, three, or ten lines from your traces. 3 and 10 lines display are here to give you more insights on the error.stack attribute.

  • With one line displayed:

  • With three lines displayed:

  • With ten lines displayed:


A Facet displays all the distinct values of an attribute or a tag as well as provides some basic analytics such as the amount of traces represented. This is also a switch to filter your data.

Facets allow you to pivot or filter your datasets based on a given attribute. Examples Facets may include users, services, etc…

Quantitative facets: measures

Use measures when you need to: * Aggregate values from multiple traces. For example, create a measure on the number of rows in Cassandra and view the P95 or top-most referrers per sum of file size requested. * Numerically compute the highest latency services for shopping cart values over $1000. * Filter continuous values. For example, the size in bytes of each payload chunk of a video stream.


Measures come with either a (long) integer or double value, for equivalent capabilities.


Measures support units (time in seconds or size in bytes) for handling of orders of magnitude at query time and display time. Unit is a property of the measure itself, not of the field. For example, consider a duration measure in nanoseconds: you have a span tag from service:A where duration:1000 stands for 1000 milliseconds, and another span tags from service:B where duration:500 stands for 500 microseconds: Scale duration into nanoseconds for all span tags flowing in with the arithmetic processor. Use a *1000000 multiplier on span tags from service:A, and a *1000 multiplier on span tags from service:B. Use duration:>20ms (see search syntax for reference) to consistently query span tags from both services at once, and see an aggregated result of max one minute.

Create a Facet

To start using an attribute as a Facet or in the search, click on it and add it as a Facet:

Once this is done, the value of this attribute is stored for all new traces and can be used in the search bar, the Facet Panel, and in the Trace graph query.

Facet Panel

Use Facets to filter on your Traces. The search bar and url automatically reflect your selections.

Further Reading