The Service Map for APM is here!

Trace Search

Overview

Use Trace Search & Analytics to filter application performance metrics and APM Events by user-defined tags. It allows deep exploration of the web requests flowing through your service.

Trace Search & Analytics can be enabled per APM service and per host. A service on which it is enabled exposes all its APM Events to Datadog.

Downstream services like databases and cache layers aren’t in the list of available services (as they don’t generate traces on their own), but their information is picked up by the top level services that call them.

In the Trace Search view you can:

APM Events

An APM event is generated every time a trace is generated. It corresponds to all the tags associated with the trace, plus the top span of the trace.

APM events aren’t just traces: traces get sampled and APM events don’t, and Datadog only keeps the top span information for the APM events, not the full trace.

APM Events can be enriched with tags, like customer, service, country, billing plan, request duration, or product type. You can then filter and query on those tags in the Trace Search & Analytics UI.

Refer to the tagging section to learn how to assign tags to a trace.

Complete traces

Trace list

If checked, APM Events listed in the trace stream have a trace associated with them, so you can display the full trace with all its associated spans.

All search parameters are contained in the url of the page, so it is very simple to share your view.

Search syntax

A query is composed of terms and operators.

There are two types of terms:

To combine multiple terms into a complex query, use any of the following boolean operators:

Operator **Description ** **Example **
AND Intersection: both terms are in the selected events (if nothing is added, AND is taken by default) authentication AND failure
OR Union: either terms is contained in the selected events authentication OR password
- Exclusion: the following term is NOT in the event authentication AND -password

To search on a specific facet you must add it as a facet first then add @ to specify you are searching on a facet.

For instance, if your facet name is url and you want to filter on the url value www.datadoghq.com just enter:

@url:www.datadoghq.com

Your traces inherit tags from hosts and integrations that generate them. They can be used in the search and as facets as well:

Query Match
("env:prod" OR test) All traces with the tag #env:prod or the tag #test
(service:srvA OR service:srvB) or (service:(srvA OR srvB)) All traces that contain tags #service:srvA or #service:srvB.
("env:prod" AND -"version:beta") All traces that contain #env:prod and that do not contain #version:beta

If your tags don’t follow tags best practices and don’t use the key:value syntax, use this search query:

  • tags:<MY_TAG>

Wildcards

To perform a multi-character wildcard search, use the * symbol as follows:

  • service:web* matches every trace that has a services starting with web
  • @url:data* matches every trace that has a url starting by data.

Numerical values

Use <,>, <=, or >= to perform a search on numerical attributes. For instance, retrieve all traces that have a response time over 100ms with:

@http.response_time:>100

It is also possible to search for numerical attributes within a specific range. For instance, retrieve all your 4xx errors with:

@http.status_code:[400 TO 499]

Autocomplete

Typing a complex query can be cumbersome. Use the search bar’s autocomplete feature to complete your query using existing values:

search bar autocomplete

Escaping of special characters

The following attributes are considered as special: ?, >, <, :, =,", ~, /, and \ require escaping. For instance, to search traces that contain user=12345 in their url the following search must be entered:

@url:*user\=JaneDoe

The same logic must be applied to spaces within trace attributes. It is not recommended to have spaces in trace attributes but in such cases, spaces require escaping. If an attribute is called user.first name, perform a search on this attribute by escaping the space:

@user.first\ name:myvalue

Saved Searches

Don’t lose time building the same views everyday. Saved searches contain your search query, columns, and time horizon. They are then available in the search bar thanks to the auto-complete matching whether the search name or query.

Saved Search

To delete a saved search, click on the bin icon under the Trace search drop-down.

Time Range

The time range allows you to display traces within a given time period. Quickly change the time range by selecting a preset range from the dropdown:

Timerange

Trace Stream

The Trace Stream is the list of traces that match the selected context. A context is defined by a search bar filter and a time range.

Sort the list by clicking the date column header.

Trace list

Displaying a full Trace

Click on any trace to see more details about it:

Trace in tracestream

Columns

To add more Trace details to the list, click the Columns button and select any Facets you want to see:

Trace list with columns

Multi-line display

Multi-line display

Choose to display one, three, or ten lines from your traces. 3 and 10 lines display are here to give you more insights on the error.stack attribute.

  • With one line displayed:

    1 line Multi-line display

  • With three lines displayed:

    2 lines with Multi-line display

  • With ten lines displayed:

    10 lines with Multi-line display

Facets

A Facet displays all the distinct values of an attribute or a tag as well as provides some basic analytics such as the amount of traces represented. This is also a switch to easily filter your data.

Facets allow you to pivot or filter your datasets based on a given attribute. Examples Facets may include users, services, etc…

Facets demo

Create a Facet

To start using an attribute as a Facet or in the search, click on it and add it as a Facet:

Create Facet

Once this is done, the value of this attribute is stored for all new traces and can be used in the search bar, the Facet Panel, and in the Trace graph query.

Facet Panel

Use Facets to easily filters on your Traces. The search bar and url automatically reflect your selections.

Facet panel

Measures

A Measure is a attribute with numerical value contained in your traces. Think of it as a “trace metric”.

Create a Measure

To start using an attribute as a measure, click on a numerical attribute of your trace:

Create a measure

Once this is done, the value of this attribute is stored for all new traces and can be used in the search bar, the Facet Panel, and in the Trace graph query.

Select the Measure Unit

All measure have their own unit that is then used for display in the Trace search columns, Trace stream widgets in dashboards, and in the Trace Graphs.

Edit a measure

Further Reading