Watchdog Insights

Error outliers

Error outliers display fields such as faceted tags or attributes containing characteristics of errors that match the current query. Statistically overrepresented key:value pairs among errors provide hints into the root causes of problems.

Typical examples of error outliers include env:staging, docker_image:acme:3.1, and http.useragent_details.browser.family:curl.

In the banner card view, you can see:

• The field name
• The proportion of errors and overall logs that the field contributes to

In the full side panel view, you can see:

• The timeseries of error logs that contain the field
• Tags that are often associated with the error logs
• A comprehensive list of log patterns

APM outliers are available on all APM pages where the Watchdog Insights carousel is available:

• Trace Explorer
• Service Page
• Resource Page

Error outliers

Error outliers display fields such as tags containing characteristics of errors that match the current query. Statistically overrepresented key:value pairs among errors provide hints into the root cause of problems.

Typical examples of error outliers include env:staging, availability_zone:us-east-1a, cluster_name:chinook, and version:v123456.

In the banner card view, you can see:

• The field name
• The proportion of errors and overall traces that the field contributes to

In the full side panel view, you can see:

• The timeseries of error traces that contain the field
• Tags that are often associated with the error traces
• A comprehensive list of related Error Tracking Issues and failing spans

Latency outliers

Latency outliers display fields such as tags that are associated with performance bottlenecks that match the current search query. key:value pairs with worse performance than the baseline can provide hints into the performance bottlenecks among a subset of APM spans.

Latency outliers are computed for the span duration.

In the banner card view, you can see:

• The field name
• The latency distribution for spans containing the tag and the baseline for the rest of the data
• A percentile of interest latency value for the outlier tag and the difference with the baseline for the rest of the data

In the full side panel, you can see a latency distribution graph for the tag and the baseline. The X axis has increments of p50, p75, p99, and max, along with a list of APM events that contain the field.

Lock pressure outlier

In the banner card view, you can see:

• The name of the impacted service
• The number of threads impacted
• The potential CPU savings (and estimated cost savings)

In the full side panel, you can see instructions on how to resolve the lock pressure:

Garbage collection outlier

In the banner card view, you can see:

• The name of the impacted service
• The amount of CPU time used to perform garbage collection

In the full side panel, you can see instructions on how to better configure garbage collection to free up some CPU time:

Regex compilation outlier

In the banner card view, you can see:

• The name of the impacted service
• The amount of CPU time spent on compiling regexes

In the full side panel, you can see instructions on how to improve regex compilation time, as well as examples of functions within your code that could be improved:

For Database Monitoring, Watchdog surfaces insights on the following metrics:

• CPU
• Commits
• IO
• Background
• Concurrency
• Idle

Find the databases impacted by one or multiple outliers by using the Insight carousel.

An overlay is then set on the databases, with pink pills highlighting the different Insights and giving more information about what happened.

Error outlier

Error outliers display fields such as faceted tags or attributes that contain characteristics of errors that match the current search query. Statistically overrepresented key:value pairs among errors can provide hints into the root causes of issues. Typical examples of error outliers include env:staging, version:1234, and browser.name:Chrome.

In the banner card view, you can see:

• The field name
• The proportion of total errors and overall RUM events that the field contributes to
• Related tags

In the full side panel, you can see a timeseries graph about the total number of RUM errors with the field, along with impact pie charts and a list of RUM events that contain the field.

Latency outlier

Latency outliers display fields such as faceted tags or attributes that are associated with performance bottlenecks that match the current search query. key:value pairs with worse performance than the baseline can provide hints into the performance bottlenecks among a subset of real users.

Latency outliers are computed for Core Web Vitals such as First Contentful Paint, First Input Delay, Cumulative Layout Shift, and Loading Time. For more information, see Monitoring Page Performance.

In the banner card view, you can see:

• The field name
• The performance metric value containing the field and the baseline for the rest of the data

In the full side panel, you can see a timeseries graph about the performance metric. The X axis has increments of p50, p75, p99, and max, along with a list of RUM events that contain the field.

For serverless infrastructures, Watchdog surfaces the following insights:

• Cold Start Ratio Up/Down
• Error Invocation Ratio Up/Down
• Memory Usage Up/Down
• OOM Ratio Up/Down
• Estimated Cost Up/Down
• Init Duration Up/Down
• Runtime Duration Up/Down

Find the serverless functions impacted by one or multiple outliers by using the Insights carousel.

An overlay is then set on the function, with pink pills highlighting the different insights and giving more information about what happened.

For Process Explorer, the Watchdog Insight carousel reflects all Process anomalies for the current context of the Process Explorer.

For Kubernetes Explorer, the Watchdog Insight carousel reflects all the Kubernetes anomalies for the current context of the Kubernetes Explorer.