How does weighted() work?

Every metrics query has a standard order of evaluation (see the Anatomy of a query for a quick review). For example, the following query is calculated as follows: sum:kubernetes.cpu.requests{*} by {kube_container_name}.rollup(avg, 10)

  1. Time aggregation – Sum the values for each timeseries (defined by a unique tag value combination) in time for each 10s rollup time interval. The number of unique tag value combinations is determined by the most volatile / high granularity tag, let’s say container_id, on this metric.
  2. Then, per kube_container_name (space aggregation), take the sum of all averaged values as a single representative value. The summed values for each kube_container_name is dependent upon the number of unique container_ids there are for each rollup interval.

The weighted() function accounts for the short lifespan of the container_id tag values when summing by kube_container_name for this gauge metric.

Example

Consider this query with the following assumptions:
sum:kubernetes_state.pod.uptime{*} by {version}.rollup(avg, 10)

  • The gauge metric’s submission interval is defined at 10 seconds.
  • A datapoint is graphed every 60 seconds in time.
  • There is a Kubernetes pod with 2 versions at any given time. Each version is labeled with an app and there is only ever 1 version per app.

The raw data over 60 seconds could resemble:

Time0s10s20s30s40s50s
app:a, version:112NANNANNANNANNAN
app:b, version:1NAN121212NANNAN
app:c, version:1NANNANNANNAN1212
app:d, version:212NANNANNANNANNAN
app:e, version:2NAN161616NANNAN
app:f, version:2NANNANNANNAN1818
  1. Time Aggregation – Rolling up data With time aggregation, we’re rolling up data either avg (without weighted) or the proposed weighted average:

    Time aggregation.rollup(avg)With .weighted()
    app:a, version:1122.0
    app:b, version:1126.0
    app:c, version:1124.0
    app:d, version:2122.0
    app:e, version:2168.0
    app:f, version:2186.0
  2. Space Aggregation Finally, the metric is aggregated by version to get the final values below:

    Space aggregation by version.rollup(avg)With .weighted()
    version:13612
    version:24616

The weighted() function remedies any inconsistent behavior with short-lived tags by weighing the values against their submission rate

Further reading

Additional helpful documentation, links, and articles: