In Datadog, metric data is ingested and stored as data points with a value and timestamp:
[ 17.82, 22:11:01 ]
A sequence of data points is stored as a time series:
[ 17.82, 22:11:01 ] [ 6.38, 22:11:12 ] [ 2.87, 22:11:38 ] [ 7.06, 22:12:00 ]
Datadog stores metric points at a 1 second resolution. However, it is recommended that you only submit points every 15 seconds. Any metrics with fractions of a second timestamps are rounded to the nearest second. If any points have the same timestamp, the latest point overwrites the previous ones.
A query extracts a stored time series and reports the data points over a defined span of time. This is a graphed time series over 15 minutes:
When the selected time span is small, all data points are displayed. However, as this time span becomes larger, it is not possible to display thousands of raw data points in a single pixel.
Datadog uses time aggregation to solve the display problem. Data points are placed into buckets of time with preset start and end points. For example, when examining four hours, data points are combined into five-minute buckets. Combining data points in this way is referred to as a rollup:
Datadog tries to return about 150 points for any given time window. Granularity becomes coarser as the amount of time requested increases. Time aggregation is done through averages.
Time series are often combined together to produce a single representative time series. For example, you might want to see the average data received by the web servers in your infrastructure.
Take two hosts submitting the same metric to Datadog:
When you look at the data separated by host,
net.bytes_rcvd is submitted at slightly different times:
To combine the two time series, the data must be time-synced. Datadog uses one of the following methods:
If no time aggregation is applied, the data points must be interpolated. A common timestamp must be decided, then the value for each time series is estimated at that time.
If time aggregation is applied, a rollup is used to create time buckets that share start and end points for each time series:
Once the points are time-aligned, the time series is space aggregated to produce a single time series representing the average of both:
In Datadog, the metric query looks like this:
Looking at the JSON, the query can be broken out by space aggregation, metric name, scope, and grouping: