Overview

Patterns automatically clusters your LLM application’s production traffic into meaningful topics, helping you understand what users are asking, identify coverage gaps, and diagnose failure modes.

You can create multiple named Patterns, each scoped to a different application, span type, or use case.

How it works

Patterns uses a mix of calls to your connected LLM provider account and text embeddings to give you an interpretable view of production behavior without manual tagging.

When you run a Pattern, it:

1. Pulls LLM interactions from your production traffic based on your filter and sampling configuration
2. Summarizes each interaction with AI-generated text
3. Computes text embedding of these summaries using a self-hosted, open source model
4. Forms clusters using machine learning (UMAP and HDBSCAN)
5. Reviews each cluster and generates meaningful topics with AI-generated text
6. Attributes each interaction to a single topic
7. Builds a hierarchy using AI by grouping similar topics together

Each topic shows its interaction volume and share of total traffic. Interactions that don’t fit any cluster are collected into an Outliers group.

Set up a Pattern

1. In Datadog, navigate to AI Observability > Agent Observability > Patterns.
2. Click + New Pattern.
3. Enter a Name.
4. Use the Application multi-selector to choose one or more LLM applications to include spans for. Selecting applications automatically updates the underlying span filter query, and editing the query updates the selected applications.
5. Under Setup:
6. Select what to Analyze with Patterns, individual Spans or Traces.
7. (Optional) Write a short Clustering Instruction to provide free-form guidance to how Patterns should group interactions and generate topics, such as user intent, agent task or product used. For example, to cluster based on the user intent, you can write “Cluster based on the user intent”. Changing this field after a first successful run will delete the topics found and persisted in previous Patterns runs. Leave this blank to group interactions by each interaction’s main goal.
8. Select the variables in your interaction you want Patterns to Cluster on. Use {{variable}} syntax to reference any span field; for example, {{meta.input.value}} to analyze patterns by user input, or {{meta.span.kind}} to analyze by span kind.
9. Add precise Filters to better select the interactions you want to cluster with Patterns.
10. Set the Time window to define the lookback period for interactions to analyze.
11. Sampling of the matched interactions is accessible by clicking the Change button. It is capped at 10 000 spans and 5 000 traces. If your filter matches more than that, Agent Observability randomly samples interactions until it reaches that number.
12. Click Select a model. The Model configuration window opens, where you can add details that Agent Observability uses to generate topic names, summaries, topic hierarchy, and to attribute each interaction to a topic: - LLM Provider: Supported providers are OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, and Vertex AI. - Account - Model: Prefer using Recommended models tagged, for performances and costs requirements. Click Confirm to save your changes and close the window.
13. (Optional) Under Datasets:
14. Turn on the radio button to compare your production traffic with your offline evaluation datasets.
15. Use the Datasets selector to select dataset(s) you use to evaluate your AI Agent offline.
16. (Optional) Enable the Automatic curation radio button to automatically fill coverage gaps Patterns detects. When enabled, Datadog creates a managed project (Patterns-coverage) and a per-pattern dataset ({pattern-name}-pattern-curated) to receive suggested interactions after each run. The toggle is on by default for new Patterns.
17. Under Schedule, choose how often Pattern runs. Scheduled times use your Datadog timezone preference. Scheduled runs use the same pipeline as a manual run, so results appear in the same place, and the Patterns page always shows your most recent run.
    • Daily, Weekdays, or Weekly (Default): Run automatically at a time (and, for weekly, a day) you choose.
    • On demand: Run the Pattern manually.
    • Custom: Run automatically every 1 to 7 days.
18. Under Notifications Enter mails and/or external messaging integrations you enabled in Datadog (see Integrations to enable external integrations in Datadog) to be notified when Patterns results are available.
19. Click Create and Run Pattern, or Create Pattern to create it without running it.

Explore your Patterns

Use the dropdown in the header to switch between your named Patterns. Each Pattern shows results from its most recent run.

Read the summary metrics

The top of the Patterns page shows three metrics from your most recent run:

• Total interactions: How many interactions were analyzed
• Identified topics: The total number of distinct topics found, including parent and child topics
• Classified: The percentage of analyzed interactions assigned to a named topic — interactions in Outliers count as unclassified

Visualize patterns by dimension

Above the topic table, a scatter plot compares your patterns against each other. Each bubble represents one topic, with the Y axis showing the number of interactions and the X axis showing the metric selected in the Dimension dropdown (for example, total errors). Use this chart to spot outliers — topics with unexpectedly high error rates or latency relative to their volume.

Navigate the topic list

The topic table provides a hierarchical view of all discovered topics. Each topic shows:

• Pattern — auto-generated name and description based on the interactions in the cluster
• Interactions — count and percentage of total traffic
• Cost — estimated LLM cost for interactions in this topic
• Tokens — token usage for interactions in this topic
• Errors — error count and rate
• Latency — median latency for interactions in this topic
• Online Evals — evaluation results if online evaluations are configured

Expand parent topics to see their sub-topics and examine specific areas of your application’s traffic.

Drill into a topic

Click any topic name to open its detail view. Use the breadcrumbs or click a node in the Topic Map to navigate the topic hierarchy. The map highlights the selected branch and shows each topic’s interaction count and, when available, traffic change since the previous run.

The detail view includes:

• Pattern Summary: Highlights traffic trends, online evaluation health, operational outliers, dataset coverage gaps, and recommended next steps.
• Scorecard: Summarizes traffic volume and share, cost, error rate, median latency, and dataset coverage.
• Dataset Coverage Detail: Shows coverage status and attribution by dataset. Review recommended cases to close coverage gaps, or open the managed dataset when automatic dataset curation is enabled.
• Evaluations: Summarizes each online evaluation with its key result and a chart over time. Open matching spans to investigate specific results.
• Interactions: Lists the inputs and outputs assigned to the topic, along with their sub-topic, timestamp, recommendation status, and evaluation results. Filter with a full-text or facet query, configure evaluation columns, or click a row to open its trace details.

Export and act on interactions

Select one or more rows in the interactions table to act on those interactions:

• Download as CSV: Export the interactions as a CSV file.
• Add to Dataset: Send the interactions to a Dataset to build evaluation test cases from real production traffic.
• Add to Queue: Send the interactions to an Annotation Queue for human review and labeling.

Trigger a new run

To analyze your production traffic, click Run analysis in the Patterns header. The pipeline runs in the background and displays it’s completed, running and pending steps. A run takes around 5 to 10 minutes. Click on Notify me to be notified by mail once the pipeline finishes. You can close the page and return later — the header shows the last run date and lookback period when the run completes.

If a run fails, a modal explains the cause and what action to take. The page continues to display results from the most recent successful run while the failed run is shown in the header.

Use topics to improve your application

Understand your production traffic

Use the topic list to see what users are actually doing with your application.

Use traffic percentage to identify your most common use cases. The parent-child hierarchy helps you move from a high-level pattern down to the specific sub-patterns underneath.

Find evaluation coverage gaps

Compare your topic distribution against what your golden datasets actually cover. Look at topics that represent high production volume but have no corresponding evaluation cases: this is where your test coverage has gaps, and where model regressions are least likely to be caught before they reach users.

Automatically curate evaluation datasets

When automatic dataset curation is enabled, each Patterns run adds suggested interactions for under-covered topics directly into a managed dataset ({pattern-name}-pattern-curated inside the Patterns-coverage project). After a run completes, open a topic’s detail view and click Access dataset to review the curated records and use them as evaluation test cases.

Diagnose failure patterns

Scope your Pattern’s filter to spans with poor quality scores or failed evaluations, then run the analysis. The resulting topic taxonomy shows which types of requests are failing most, giving you a structured way to prioritize fixes instead of debugging trace by trace.

Track how traffic evolves

Re-run your Pattern periodically and use the Compare to dropdown to compare topic distributions across runs. When a topic marked NEW appears near the top, that signals your users have found a new use case or a new failure mode.

Further Reading

Additional helpful documentation, links, and articles:

Learn about Agent ObservabilityDOCUMENTATION Learn about Agent Observability Key Terms and ConceptsDOCUMENTATION Learn about DatasetsDOCUMENTATION Investigate with LLM ObservabilityLEARNING CENTER Tracing LLM ApplicationsLEARNING CENTER Understand production LLM behavior with Patterns in Agent ObservabilityBLOG