---
title: Using USM Metrics in Monitors, SLOs, and Dashboards
description: Learn how to create monitors, SLOs, and dashboards using your USM metrics.
breadcrumbs: >-
  Docs > Universal Service Monitoring > Universal Service Monitoring Guides >
  Using USM Metrics in Monitors, SLOs, and Dashboards
---

# Using USM Metrics in Monitors, SLOs, and Dashboards

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). ().
{% /alert %}

{% /callout %}

## Overview{% #overview %}

[Universal Service Monitoring](https://docs.datadoghq.com/universal_service_monitoring) discovers services using popular container tags (such as `app`, `short_image`, and `kube_deployment`) and generates entries in the [Software Catalog](https://docs.datadoghq.com/tracing/software_catalog) for those services.

You can access request, error, and duration metrics in Datadog for both inbound and outbound traffic on all services discovered with Universal Service Monitoring. These service health metrics are useful for creating alerts, [tracking deployments](https://docs.datadoghq.com/tracing/services/deployment_tracking/), and getting started with [service level objectives (SLOs)](https://docs.datadoghq.com/service_level_objectives) so you can get broad visibility into all services running on your infrastructure.

{% image
   source="https://datadog-docs.imgix.net/images/universal_service_monitoring/guide/usm_slo.241abfde9c8c68aaf2565ebf94d9b666.png?auto=format"
   alt="Universal Service Monitoring SLOs for BITSBOUTIQUE" /%}

This guide describes how to search for USM metrics such as `universal.http.*` and use them in your monitors, SLOs, and dashboards.

## USM metrics vs APM metrics{% #usm-metrics-vs-apm-metrics %}

| Metric Name                 | Units   | Type         | Description                                            |
| --------------------------- | ------- | ------------ | ------------------------------------------------------ |
| universal.http.client       | Seconds | Distribution | Outbound request latency, counts, errors, and rates.   |
| universal.http.client.hits  | Hits    | Count        | Total number of outbound requests and errors.          |
| universal.http.client.apdex | Score   | Gauge        | The Apdex score of outbound requests for this service. |
| universal.http.server       | Seconds | Distribution | Inbound request latency, counts, errors, and rates.    |
| universal.http.server.hits  | Hits    | Count        | Total number of inbound requests and errors.           |
| universal.http.server.apdex | Score   | Gauge        | The Apdex score for this web service.                  |

Unlike APM metrics, errors are available under the `error:true` tag instead of as a separate metric.

**Note:** The `.hits` metrics have all of your infrastructure tags and are the recommended way to query request and error counts. You can also add [second primary tags](https://docs.datadoghq.com/tracing/guide/setting_primary_tags_to_scope/?tab=helm#add-a-second-primary-tag-in-datadog) to all USM metrics.

### Metric syntax{% #metric-syntax %}

The USM metric query syntax differs from the [APM metric query syntax](https://docs.datadoghq.com/tracing/metrics/metrics_namespace), which uses `trace.*`. USM Metrics fall under a single distribution metric name.

For example:

| APM                                             | USM                                                  |
| ----------------------------------------------- | ---------------------------------------------------- |
| trace.universal.http.client.hits{*}             | count:universal.http.client{*}                       |
| trace.universal.http.client.errors              | count:universal.http.client{error:true}              |
| trace.universal.http.client.hits.by_http_status | count:universal.http.client{*} by http_status_family |
| pXX:trace.universal.http.client{*}              | pXX:universal.http.client{*}                         |
| trace.universal.http.client.apdex{*}            | universal.http.client.apdex{*}                       |

The same translations apply for the `universal.http.server` operation that captures inbound traffic. For more information about distribution metrics, see [DDSketch-based Metrics in APM](https://docs.datadoghq.com/tracing/guide/ddsketch_trace_metrics/).

## Usage{% #usage %}

Navigate to [**Infrastructure > Universal Service Monitoring**](https://app.datadoghq.com/services), filter by Universal Service Monitoring telemetry type, and click on a service. The **Performance** tab displays service-level graphs on hits, latency, requests, errors, and more. You can also access these metrics when creating a monitor or an SLO, or by looking at a dashboard in the [Software Catalog](https://docs.datadoghq.com/tracing/software_catalog).

### Create a monitor{% #create-a-monitor %}

You can create an [**APM Monitor**](https://app.datadoghq.com/monitors/create/apm) to trigger an alert when a USM metric such as `universal.http.client` either crosses a threshold or deviates from an expected pattern.

1. Navigate to [**Monitors > New Monitor**](https://app.datadoghq.com/monitors/create) and click [**APM**](https://app.datadoghq.com/monitors/create/apm).
1. Select **APM Metrics** and define a service or resource's `env` and any other [primary tags](https://docs.datadoghq.com/metrics/advanced-filtering/). Select a service or resource to monitor and define time interval for the monitor to evaluate the query over.
1. Select **Threshold Alert** and select a USM metric such as `Requests per Second` for the monitor to trigger on. Then, define if the value should be **above** or **below** the alert and warning thresholds. Enter a value for the alert threshold, and optionally, for the warning threshold.
1. The notification section contains a prepopulated message for the monitor. Customize the alert name and message and define the permissions for this monitor.
1. Click **Create**.

{% image
   source="https://datadog-docs.imgix.net/images/universal_service_monitoring/guide/usm_monitor.be7e5939082ad299adf303465e249bef.png?auto=format"
   alt="Universal Service Monitoring Monitor for BITSBOUTIQUE" /%}

For more information, see the [APM Monitor documentation](https://docs.datadoghq.com/monitors/create/types/apm).

### Create an SLO{% #create-an-slo %}

You can create an [**SLO**](https://app.datadoghq.com/slo/new) on a per-service basis to ensure you are meeting objectives set by USM metrics and improving availability over time. Datadog recommends [creating an SLO programmatically](https://docs.datadoghq.com/api/latest/service-level-objectives/) to cover a lot of services.

To create an SLO from the Software Catalog:

1. Navigate to the **Reliability** tab of the [Software Catalog](https://app.datadoghq.com/services).
1. Under the **SLOs** column, hover over a service and click **+ Create Availability SLO** or **+ Create Latency SLO**.

{% image
   source="?auto=format"
   alt="Setting up a Universal Service Monitoring SLO for BITSBOUTIQUE" /%}

Optionally, to create an SLO manually using USM metrics:

1. Navigate to [**SLOs**](https://app.datadoghq.com/slo/manage) and click [**New SLO**](https://app.datadoghq.com/slo/new).

1. Select **Metric Based** and create two queries in the **Good events (numerator)** section:

   - Query A: Enter a USM metric such as `universal.http.server`, filter to a specific service by adding primary `service` and `env` tags in the `from` field, and select `count` in the `as` field.
   - Query B: Enter a USM metric such as `universal.http.server`, filter to a specific service by adding primary `service` and `env` tags, in addition to an `error:true` tag in the `from` field, and select `count` in the `as` field.

1. Click **+ Add Formula** and enter `a-b`.

1. In the **Total events (denominator)** section, enter a USM metric such as `universal.http.server`, filter to a specific service by adding primary `service` and `env` tags in the `from` field, and select `count` in the `as` field.

1. Click **+ New Target** to create a target threshold with the following settings:

   - The time window is `7 Days`, the target threshold is `95%`, and the warning threshold is `99.5%`. Datadog recommends setting the same target threshold across all time windows.

1. Enter a name and description for this SLO. Set primary `env` and `service` tags, in addition to the `team` tag.

1. Click **Save and Set Alert**.

{% image
   source="https://datadog-docs.imgix.net/images/universal_service_monitoring/guide/usm_slo_setup.ce374fd0c408193aaec7e9a27fd72c46.png?auto=format"
   alt="Setting up a Universal Service Monitoring SLO for BITSBOUTIQUE" /%}

For more information, see the [Service Level Objectives documentation](https://docs.datadoghq.com/service_level_objectives/).

### Access a defined dashboard{% #access-a-defined-dashboard %}

The [Software Catalog](https://docs.datadoghq.com/tracing/software_catalog) identifies dashboards defined in your service definition file and lists them on the **Dashboards** tab. Click **Manage Dashboards** to access and edit the service definition directly in GitHub.

{% image
   source="https://datadog-docs.imgix.net/images/universal_service_monitoring/guide/manage_dashboards.79521ed27e36b36dd338eb2229885825.png?auto=format"
   alt="Manage Dashboards button in the Dashboards tab of a service in the Software Catalog" /%}

For more information, see the [Dashboards documentation](https://docs.datadoghq.com/dashboards).

## Further Reading{% #further-reading %}

- [Automatically discover, map, and monitor all your services in seconds with Universal Service Monitoring](https://www.datadoghq.com/blog/universal-service-monitoring-datadog/)
- [Learn about Universal Service Monitoring](https://docs.datadoghq.com/universal_service_monitoring)
- [Learn about APM Metrics](https://docs.datadoghq.com/tracing/metrics)