New announcements for Serverless, Network, RUM, and more from Dash!

Metrics

Introduction

This section explains what metrics are and what they do. Whether you want to send custom metrics, or have a better understanding about how Datadog works, read on. If you’re looking for information about the DogStatsD (which implements these metrics), see the DogStatsD documentation.

Submitting metrics

There are multiple ways to send metrics to Datadog:

  1. Via the Datadog Agent directly. Learn how to write an integration, or examine the Aggregator source code directly.
  2. Via the DogStatsD server (bundled with the Datadog Agent) and a client library.
  3. Directly via Datadog’s HTTP API.
  4. Via the Dropwizard Java metrics library with the metrics-datadog backend. Thanks to the people at Vistar Media, Coursera, and Bazaarvoice for their contributions.
Metric timestamps cannot be more than 10 minutes in the future or more than 1 hour in the past.

Naming metrics

There are a few rules regarding metric names:

  • Must start with a letter.
  • Must only contain ASCII alphanumerics, underscores, and periods.
    • Other characters, including spaces, are converted to underscores.
    • Unicode is not supported.
  • Must not exceed 200 characters. Fewer than 100 is preferred from a UI perspective.

Metrics reported by the Agent are in a pseudo-hierarchical dotted format (e.g. http.nginx.response_time). The hierarchy is neither enforced nor interpreted, but it can be used to infer things about servers. For example, if hostA and hostB are both reporting http.nginx.* those must be web frontends.

Note: Metric names are case sensitive in Datadog.

Metric Types

The “Datadog in-app type” affects how a given metric is interpreted in query results and graph visualizations across the application. This type is visible and can be changed on the metric summary page. Be aware that changing the metric type may render historical data nonsensical.

In the Datadog web application there are four metric types (though one is deprecated):

  • COUNT
  • COUNTER (deprecated)
  • GAUGE
  • RATE

A metric’s type is stored as metrics metadata and is used to determine how a metric is interpreted throughout the application by determining default time aggregation function and as_rate()/as_count() behavior. The as_count() and as_rate() modifiers behave differently for different web application metric types.

Submission types and Datadog in-app types

Datadog accepts metrics submitted from a variety of sources, and as a result the “submission type” (think “use-case”) does not always map exactly to the Datadog in-app type:

Submission SourceSubmission Method (python)Submission TypeDatadog In-App Type
APIapi.Metric.send(type="count", ...)countcount
APIapi.Metric.send(type="gauge", ...)gaugegauge
APIapi.Metric.send(type="rate", ...)raterate
DogStatsDdog.gauge(...)gaugegauge
DogStatsDdog.histogram(...)histogramgauge, rate
DogStatsDdog.increment(...)counterrate
DogStatsDdog.set(...)setgauge
Agent checkself.count(...)countcount
Agent checkself.gauge(...)gaugegauge
Agent checkself.histogram(...)histogramgauge, rate
Agent checkself.increment(...)counter deprecatedrate
Agent checkself.monotonic_count(...)monotonic_countcount
Agent checkself.rate(...)rategauge
Agent checkself.set(...)setgauge

Modify a metric’s type

While it is not normally required, it is possible to change a metric’s type. For example:

  1. You have a metric app.requests.served that counts requests served, but accidentally submitted it via StatsD as a gauge. The metric’s Datadog type is therefore gauge.

  2. You wanted to submit app.requests.served as a StatsD count metric for time aggregation. This would help answer questions like “How many total requests were served in the past day?” by querying sum:app.requests.served{*} (this would not make sense for a gauge-type metric.)

  3. You like the name app.requests.served so rather than submitting a new metric name with the more appropriate count type, you could change the type of app.requests.served by updating:

    • Your submission code, calling dogstatsd.increment('app.requests.served', N) after N requests are served.
    • The Datadog in-app type via the metric summary page to rate.

This causes data submitted before the type change for app.requests.servedto behave incorrectly because it was stored in a format to be interpreted as an in-app gauge not a rate. Data submitted after steps 3a and 3b is interpreted properly.

If you are not willing to lose the historical data submitted as a gauge, create a new metric name with the new type, leaving the type of app.requests.served unchanged.

Note: For the AgentCheck, self.increment does not calculate the delta for a monotonically increasing counter, rather it reports the value passed in at the check run. To send the delta value on a monotonically increasing counter, use self.monotonic_count.

Units

To eliminate ambiguity and help you make sense of your systems, the following units may be associated with metrics submitted to Datadog.

typeunit(s)
BYTESbit / byte / kibibyte / mebibyte / gibibyte / tebibyte / pebibyte / exbibyte
TIMEnanosecond / microsecond / millisecond / second / minute / hour / day / week
PERCENTAGEpercent_nano / percent / apdex / fraction
NETWORKconnection / request / packet / segment / response / message / payload / timeout / datagram / route / session
SYSTEMprocess / thread / host / node / fault / service / instance / cpu
DISKfile / inode / sector / block
GENERALbuffer / error / read / write / occurrence / event / time / unit / operation / item / task / worker / resource / garbage collection / email / sample / stage / monitor / location / check / attempt / device / update / method / job / container / execution / throttle / invocation / user / success / build / prediction
DBtable / index / lock / transaction / query / row / key / command / offset / record / object / cursor / assertion / scan / document / shard / flush / merge / refresh / fetch / column / commit / wait / ticket / question
CACHEhit / miss / eviction / get / set
MONEYdollar / cent
MEMORYpage / split
FREQUENCYhertz / kilohertz / megahertz / gigahertz
LOGGINGentry
TEMPERATUREdegree celsius / degree fahrenheit
CPUnanocore / microcore / millicore / core / kilocore / megacore / gigacore / teracore / petacore / exacore

Units are displayed automatically on timeseries graphs, query value widgets, and toplists, as shown in the screenshot of the Redis dashboard below:

On timeseries graphs, move your cursor over any graph to see the relevant units. The raw data is automatically converted to readable display units (fractions of a second to ms, millions of bytes per second to MiB/s, etc.):

Units are also displayed at the bottom of timeboard graphs, and metric descriptions are available by selecting Metrics Info from the gear dropdown:

Further reading