Logs can be valuable as individual events, but sometimes valuable information lives in a subset of events. In order to expose this information, group your logs by fields, identify patterns, or aggregate your logs into transactions.
Switch between different aggregations of your queried logs with the logs query editor. The fields you select to group, aggregate, and measure your logs are saved as you switch between different visualizations and aggregation types.
Add multiple queries to simultaneously analyze different sets of logs, and apply formulas and functions to your queries for in-depth analysis.
Note: Aggregations are supported for indexed logs only. If you need to perform aggregation on non-indexed logs, consider temporarily disabling exclusion filters, using logs to metrics and/or running a rehydration on your archives.
When aggregating by Fields, all logs matching your query filter are aggregated into groups based on the value of one or multiple log facets. On top of these aggregates, you can extract the following measures:
- count of logs per group
- unique count of coded values for a facet per group
- statistical operations (
percentiles) on numerical values of a facet per group
Note: Individual logs with multiple values for a single facet belong to that many aggregates. For instance, a log having the
team:sre and the
team:marketplace tags are counted once in the
team:sre aggregate and once in the
The Fields aggregation supports one dimension for the Top list visualization, and up to three dimensions for the Timeseries and Table visualizations. When there are multiple dimensions, the top values are determined according to the first dimension, then according to the second dimension within the top values of the first dimension, then according to the third dimension within the top values of the second dimension.
Multiple queries are supported in Timeseries and Top list visualizations. Add multiple queries by clicking on the
+ Add button next to the query editor. When you add a new query, it is a copy of the last query and its grouping options:
Select or deselect queries to display in the current visualization by clicking on their letters in the query editor:
By default, when a new query is added, it is automatically selected to be displayed in the chosen visualization.
Display the timeline for one of your queries by selecting that query in the
Timeline for dropdown. Scope one of your search queries by selecting that query in the
Use facets with dropdown and clicking on values in the Facet Panel. Only the selected query is updated with the chosen facets.
Note: Functions are only supported in Timeseries and Top list visualizations.
Apply functions to your logs by clicking on the
Fields aggregation in the query editor. Optionally select a faceted field to apply the function to, then click on the
Σ icon next to that measure. Select or search for a function to apply to the selected log field.
All functions available for logs in the graphing editor in Dashboards can be applied to logs in the Log Explorer:
Here is an example of how to apply an Exclusion function to exclude certain values of your logs:
Apply a formula on one or multiple queries by clicking on the
+ Add button next to the query editor. In the following example, the formula is used to calculate the ratio of the unique number of
Cart Id in logs for
Merchant Tier: Enterprise /
Merchant Tier: Premium customers:
Note: To apply formulas with multiple queries, all queries must be grouped by the same facet. In the example above, both queries are grouped by
Webstore Store Name.
You can apply a function to a formula by clicking on the
Σ icon. Here is an example of how to apply a Timeshift function on the proportion of error logs in all logs to compare current data with data from one week before:
With pattern aggregation, logs that have a
message with similar structures, belong to the same
service and have the same
status are grouped altogether. The patterns view is helpful for detecting and filtering noisy error patterns that could cause you to miss other issues:
Note: The pattern detection is based on 10,000 log samples. Refine the search to see patterns limited to a specific subset of logs.
Patterns support the List Aggregates visualization. Clicking a pattern in the list opens the pattern side panel from which you can:
- Access a sample of logs from that pattern
- Append the search filter to scope it down to logs from this pattern only
- Get a kickstart for a grok parsing rule to extract structured information logs of that pattern
Transactions aggregate indexed logs according to instances of a sequence of events, such as a user session or a request processed across multiple micro-services. For example, an e-commerce website groups logs across various user actions, such as catalog search, add to cart, and checkout, to build a transaction view using a common attribute such as
Note: The transaction aggregation differs from the natural group aggregation, in the sense that resulting aggregates not only include logs matching the query, but also all logs belonging to the related transactions.
- Duration: The difference of timestamps for the last and first log in the transaction. This measure is automatically added.
- Maximum Severity found in logs in the transaction. This measure is automatically added.
- Finding key items: For any
facet with string values, calculate specific log information using the operations
- Getting Statistics: For any
measure, calculate statistical information using the operations
Transactions support the List Aggregates visualization. Clicking a pattern in the list opens the pattern side panel from which you can:
- Access all logs within that transaction
- Search specific logs within that transaction
Additional helpful documentation, links, and articles: