Upgrade Your Filter Queries to the New Search Syntax

Docs > Observability Pipelines > Observability Pipelines Guides > Upgrade Your Filter Queries to the New Search Syntax

Overview

Worker versions 2.11 and newer use an updated search syntax. This document covers the following:

How to upgrade existing queries to the new syntax
What’s new in the updated search syntax

How to upgrade queries to the new search syntax

See the steps based on whether you:

Created the pipeline in the UI
Created the pipeline using the API or Terraform

Created the pipeline in the UI

If you created your pipeline in the UI:

Upgrade to Observability Pipelines Worker version 2.11.
Navigate to the Pipeline page for that pipeline and update your filter queries to the new syntax. See the What’s new in the updated search syntax section for more information.
On the pipeline editor page, by default the Use legacy search syntax box is checked because your pipeline is running the old search syntax of Worker 2.10 or older.
After you’ve updated all queries in that pipeline, uncheck the Use legacy search syntax box and deploy your pipeline.

Created the pipeline using the API or Terraform

If your pipeline was created using the public API or Terraform:

Within the same request that you make to update your pipeline queries to the new search syntax, set use_legacy_search_syntax to false.

You must set use_legacy_search_syntax to false when you update your queries. If use_legacy_search_syntax is unpopulated, it defaults to true in the Worker.

What’s new in the updated search syntax

The following table lists the differences between the legacy and new search syntax:

Legacy syntax	New syntax
Requires the `@` symbol for attribute search, except when referencing reserved fields.	Does not require the `@` symbol for attribute search.
Since `@` indicates an attribute search, tag searches do not include an `@`, and are matched under the attributes `tags` and `ddtags`. Attribute search queries without an `@` symbol are matched against the `tags` or `ddtags` array. Example attribute search syntax: `env:prod`	Tags syntax must be explicitly entered. Inspect your data with Live capture to determine which fields to match. Example attribute search syntax: `tags:"env:prod" OR ddtags:"env:prod"`
Reserved fields do not require the `@` symbol.	Reserved fields do not require the `@` symbol.

Note: The upgraded search syntax does not require the @ symbol for attribute searches. You do not need to remove the @ symbol from existing filter queries, but Datadog recommends that you remove the @ symbol from your queries.

The following examples show matched logs, along with the legacy syntax and new syntax that matches the logs.

{"user": "firstname.lastname"}: Legacy syntax: @user:firstname.lastname; New syntax: user:firstname.lastname; Difference: The new syntax does not require the @ symbol for attribute search.
{"message": {"log_level": "ERROR"}}: Legacy syntax: @message.log_level:ERROR; New syntax: message.log_level:ERROR; Difference: The new syntax does not require the @ symbol for attribute search.
{"status": "INFO"}: Legacy syntax: status:INFO; New syntax: status:INFO; Difference: No changes because status was previously a reserved field that could be filtered without using the @ symbol. The new syntax does not use the @ symbol for attribute searches.
{"message": "Hello, world" } {"message: "hello world"} {"message": "Hello-world"}: Legacy syntax: message:"hello world"; New syntax: message:"hello world"; Difference: There are no changes between the legacy and new syntax because message was a reserved field in the legacy search syntax and didn’t require the @ symbol. The new syntax does not use the @ symbol for attribute searches.
{"message": "hEllo world"}: Legacy syntax: HELLO OR hello OR Hello; New syntax: hello; Difference: With the new syntax, free text search is case insensitive.
{"user": "name"}: Legacy syntax: @user:(name OR Name OR nAme); New syntax: user:(name OR Name or nAme); Difference: With the new syntax, attribute search is case sensitive and the @ symbol is not required for attribute search.
{"tags": ["env:prod"] } {"ddtags": ["env:prod"] }: Legacy syntax: env:prod; New syntax: tags:"env:prod" OR ddtags:"env:prod"; Difference: With the legacy syntax, when the syntax does not contain the @ symbol and is not searching for a reserved field, all terms are matched with the tags or ddtags field. With the new search syntax, there are no reserved fields so all searches must be entered explicitly.
{"tags": ["message.log_level:INFO"] } {"ddtags": ["message.log_level:INFO"]}}: Legacy syntax: message.log_level:INFO; New syntax: tags:"message.log_level:INFO" OR ddtags:"message.log_level:INFO"; Difference: Same reason as the previous query for env:prod query.
{"source": "postgres" } {"ddsource":"postgres" }: Legacy syntax: source:postgres; New syntax: source:postgres OR ddsource:postgres; Difference: With the legacy syntax, attribute search with source field matches both source and ddsource fields. The new syntax no longer does this so you must enter source or ddsource explicitly.

Note: Using wildcards for field names in attribute search is not supported for either the legacy or new syntax. For example, the following wildcard usage does not work:

Legacy syntax: *:something
New syntax: *:something