---
title: Patterns
description: >-
  Discover and analyze production traffic patterns in your agent with automated
  topic clustering.
breadcrumbs: Docs > Agent Observability > Investigate > Patterns
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Patterns

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Patterns automatically clusters your LLM application's production traffic into meaningful topics, helping you understand what users are asking, identify coverage gaps, and diagnose failure modes.

You can create multiple named Patterns, each scoped to a different application, span type, or use case.

## How it works{% #how-it-works %}

Patterns uses a mix of calls to your [connected LLM provider account](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/connect_to_account.md) and text embeddings to give you an interpretable view of production behavior without manual tagging.

When you run a Pattern, it:

1. Pulls LLM interactions from your production traffic based on your filter and sampling configuration
1. Summarizes each interaction with AI-generated text
1. Computes text embedding of these summaries using a self-hosted, open source model
1. Forms clusters using machine learning (UMAP and HDBSCAN)
1. Reviews each cluster and generates meaningful topics with AI-generated text
1. Attributes each interaction to a single topic
1. Builds a hierarchy using AI by grouping similar topics together

Each topic shows its interaction volume and share of total traffic. Interactions that don't fit any cluster are collected into an Outliers group.

## Set up a Pattern{% #set-up-a-pattern %}

1. In Datadog, navigate to **AI Observability** > **Agent Observability** > [**Patterns**](https://app.datadoghq.com/llm/patterns).
1. Click **+ New Pattern**.
1. Enter a **Name**.
1. Use the **Application** multi-selector to choose one or more LLM applications to include spans for. Selecting applications automatically updates the underlying span filter query, and editing the query updates the selected applications.
1. Under **Setup**:
1. Select what to **Analyze** with Patterns, individual **Spans** or **Traces**.
1. (Optional) Write a short **Clustering Instruction** to provide free-form guidance to how Patterns should group interactions and generate topics, such as user intent, agent task or product used. For example, to cluster based on the user intent, you can write "Cluster based on the user intent". Changing this field after a first successful run **will delete the topics found and persisted** in previous Patterns runs. Leave this blank to group interactions by each interaction's main goal.
1. Select the variables in your interaction you want Patterns to **Cluster on**. Use `{{variable}}` syntax to reference any span field; for example, `{{meta.input.value}}` to analyze patterns by user input, or `{{meta.span.kind}}` to analyze by span kind.
1. Add precise **Filters** to better select the interactions you want to cluster with Patterns.
1. Set the **Time window** to define the lookback period for interactions to analyze.
1. **Sampling** of the matched interactions is accessible by clicking the **Change** button. It is capped at 10 000 spans and 5 000 traces. If your filter matches more than that, Agent Observability randomly samples interactions until it reaches that number.
1. Click **Select a model**. The Model configuration window opens, where you can add details that Agent Observability uses to generate topic names, summaries, topic hierarchy, and to attribute each interaction to a topic: - **LLM Provider**: Supported providers are OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, and Vertex AI. - **Account** - **Model**: Prefer using Recommended models tagged, for performances and costs requirements. Click **Confirm** to save your changes and close the window.
1. (Optional) Under **Datasets**:
1. Turn on the radio button to compare your production traffic with your offline evaluation datasets.
1. Use the **Datasets** selector to select dataset(s) you use to evaluate your AI Agent offline.
1. (Optional) Enable the **Automatic curation** radio button to automatically fill coverage gaps Patterns detects. When enabled, Datadog creates a managed project (`Patterns-coverage`) and a per-pattern dataset (`{pattern-name}-pattern-curated`) to receive suggested interactions after each run. The toggle is **on** by default for new Patterns.
1. Under **Schedule**, choose how often Pattern runs. Scheduled times use your Datadog timezone preference. Scheduled runs use the same pipeline as a manual run, so results appear in the same place, and the Patterns page always shows your most recent run.
   - **Daily**, **Weekdays**, or **Weekly (Default)**: Run automatically at a time (and, for weekly, a day) you choose.
   - **On demand**: Run the Pattern manually.
   - **Custom**: Run automatically every 1 to 7 days.
1. Under **Notifications** Enter mails and/or external messaging integrations you enabled in Datadog (see [**Integrations**](https://docs.datadoghq.com/integrations.md#cat-notifications) to enable external integrations in Datadog) to be notified when Patterns results are available.
1. Click **Create and Run Pattern**, or **Create Pattern** to create it without running it.

## Explore your Patterns{% #explore-your-patterns %}

Use the dropdown in the header to switch between your named Patterns. Each Pattern shows results from its most recent run.

### Read the summary metrics{% #read-the-summary-metrics %}

The top of the Patterns page shows three metrics from your most recent run:

- Total interactions: How many interactions were analyzed
- Identified topics: The total number of distinct topics found, including parent and child topics
- Classified: The percentage of analyzed interactions assigned to a named topic — interactions in Outliers count as unclassified

### Visualize patterns by dimension{% #visualize-patterns-by-dimension %}

Above the topic table, a scatter plot compares your patterns against each other. Each bubble represents one topic, with the Y axis showing the number of interactions and the X axis showing the metric selected in the Dimension dropdown (for example, total errors). Use this chart to spot outliers — topics with unexpectedly high error rates or latency relative to their volume.

{% image
   source="https://docs.dd-static.net/images/llm_observability/patterns_landing_page.25185b92a9b6adcd7219c1a6ea0043c2.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/patterns_landing_page.25185b92a9b6adcd7219c1a6ea0043c2.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Patterns page showing a bubble chart with one bubble per topic. The Y axis shows interaction count and the X axis shows the selected metric dimension." /%}

### Navigate the topic list{% #navigate-the-topic-list %}

The topic table provides a hierarchical view of all discovered topics. Each topic shows:

- Pattern — auto-generated name and description based on the interactions in the cluster
- Interactions — count and percentage of total traffic
- Cost — estimated LLM cost for interactions in this topic
- Tokens — token usage for interactions in this topic
- Errors — error count and rate
- Latency — median latency for interactions in this topic
- Online Evals — evaluation results if online evaluations are configured

Expand parent topics to see their sub-topics and examine specific areas of your application's traffic.

### Drill into a topic{% #drill-into-a-topic %}

Click any topic name to open its detail view. Use the breadcrumbs or click a node in the Topic Map to navigate the topic hierarchy. The map highlights the selected branch and shows each topic's interaction count and, when available, traffic change since the previous run.

{% image
   source="https://docs.dd-static.net/images/llm_observability/patterns_topic_details.c2956590ac13e3a2a00c4735d7d0bffa.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/patterns_topic_details.c2956590ac13e3a2a00c4735d7d0bffa.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The topic detail view showing a pattern summary and an interactive topic map with interaction counts and traffic changes." /%}

The detail view includes:

- Pattern Summary: Highlights traffic trends, online evaluation health, operational outliers, dataset coverage gaps, and recommended next steps.
- Scorecard: Summarizes traffic volume and share, cost, error rate, median latency, and dataset coverage.
- Dataset Coverage Detail: Shows coverage status and attribution by dataset. Review recommended cases to close coverage gaps, or open the managed dataset when automatic dataset curation is enabled.
- Evaluations: Summarizes each online evaluation with its key result and a chart over time. Open matching spans to investigate specific results.
- Interactions: Lists the inputs and outputs assigned to the topic, along with their sub-topic, timestamp, recommendation status, and evaluation results. Filter with a full-text or facet query, configure evaluation columns, or click a row to open its trace details.

{% image
   source="https://docs.dd-static.net/images/llm_observability/patterns_topic_details_scorecard.6367ebd3f36ac5cd2af5bdec3b8a034b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/patterns_topic_details_scorecard.6367ebd3f36ac5cd2af5bdec3b8a034b.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The topic detail view showing the scorecard, dataset coverage details, and interactions table." /%}

### Export and act on interactions{% #export-and-act-on-interactions %}

Select one or more rows in the interactions table to act on those interactions:

- **Download as CSV:** Export the interactions as a CSV file.
- **Add to Dataset:** Send the interactions to a [Dataset](https://docs.datadoghq.com/llm_observability/improve/datasets.md) to build evaluation test cases from real production traffic.
- **Add to Queue:** Send the interactions to an [Annotation Queue](https://docs.datadoghq.com/llm_observability/investigate/annotation_queues.md) for human review and labeling.

## Trigger a new run{% #trigger-a-new-run %}

To analyze your production traffic, click Run analysis in the Patterns header. The pipeline runs in the background and displays it's completed, running and pending steps. A run takes around 5 to 10 minutes. Click on **Notify me** to be notified by mail once the pipeline finishes. You can close the page and return later — the header shows the last run date and lookback period when the run completes.

If a run fails, a modal explains the cause and what action to take. The page continues to display results from the most recent successful run while the failed run is shown in the header.

## Use topics to improve your application{% #use-topics-to-improve-your-application %}

### Understand your production traffic{% #understand-your-production-traffic %}

Use the topic list to see what users are actually doing with your application.

Use traffic percentage to identify your most common use cases. The parent-child hierarchy helps you move from a high-level pattern down to the specific sub-patterns underneath.

### Find evaluation coverage gaps{% #find-evaluation-coverage-gaps %}

Compare your topic distribution against what your golden datasets actually cover. Look at topics that represent high production volume but have no corresponding evaluation cases: this is where your test coverage has gaps, and where model regressions are least likely to be caught before they reach users.

### Automatically curate evaluation datasets{% #automatically-curate-evaluation-datasets %}

When automatic dataset curation is enabled, each Patterns run adds suggested interactions for under-covered topics directly into a managed dataset (`{pattern-name}-pattern-curated` inside the `Patterns-coverage` project). After a run completes, open a topic's detail view and click **Access dataset** to review the curated records and use them as evaluation test cases.

### Diagnose failure patterns{% #diagnose-failure-patterns %}

Scope your Pattern's filter to spans with poor quality scores or failed evaluations, then run the analysis. The resulting topic taxonomy shows which types of requests are failing most, giving you a structured way to prioritize fixes instead of debugging trace by trace.

### Track how traffic evolves{% #track-how-traffic-evolves %}

Re-run your Pattern periodically and use the Compare to dropdown to compare topic distributions across runs. When a topic marked NEW appears near the top, that signals your users have found a new use case or a new failure mode.

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Learn about Agent Observability](https://docs.datadoghq.com/llm_observability.md)
- [Learn about Agent Observability Key Terms and Concepts](https://docs.datadoghq.com/llm_observability/quickstart/terms.md)
- [Learn about Datasets](https://docs.datadoghq.com/llm_observability/improve/datasets.md)
- [Investigate with LLM Observability](https://learn.datadoghq.com/courses/llm-obs-investigations)
- [Tracing LLM Applications](https://learn.datadoghq.com/courses/llm-obs-tracing-llm-applications)
- [Understand production LLM behavior with Patterns in Agent Observability](https://www.datadoghq.com/blog/patterns-agent-observability/)