Datadog allows for custom metrics submission through multiple ways to offer a full understanding of what is happening on your infrastructure.
This article explains:
A custom metric refers to a single, unique combination of a metric name, host, and any tags.
Custom metrics generally refer to any metric that you send using statsd, DogStatsD, or through extensions made to the Datadog Agent. Some integrations can potentially emit an unlimited number of metrics that can also count as custom, further details on which standard integrations emit custom metrics.
In order to fully leverage the capabilities of the Datadog product through scoping and alerting, you’ll probably be using tags. As a consequence, one submitted metric actually leads to multiple unique tag combinations- counting towards your custom metrics count.
The given unique metrics on a given host are therefore:
exception:A//unique because of new tag
In this situation, you would end up with 6 different metrics.
Note that the ordering of tags does not matter, so the following two metrics would be considered non-unique:
Datadog offers 2 plans - Pro & Enterprise. Pro customers are allotted 100 custom metrics per host & Enterprise customers are allotted 200 custom metrics per host. These are counted across your entire infrastructure rather than on a per-host basis. For example, if you were on the Pro plan and are licensed for 3 hosts, you would have 300 custom metrics by default - these 300 metrics may be divided equally amongst each individual host, or all 300 metrics could be sent from a single host.
Using the aforementioned example, below shows three scenarios which would all be acceptable without exceeding the default metric count for three hosts:
We do not enforce any fixed rate limit on custom metric submission, if you’re exceeding your default allotment, our teams will reach out to you.
When creating a custom metric, all the host tags are automatically added to that metric as one unique tag combination, to which you’ll add the tags linked to the metric itself. Those are the most important as they add to the actual metric count.
Let’s say you want to have insight into the request.count from different services across your infrastructure.
The logic behind your metric is the following :
From there, you can see that on each host reporting this metric, if all services report both successes and failures, you can have up to 1x2x3 = 6 custom metrics.
Let’s say you have 3 hosts:
Across your 3 hosts, you’d have 13 distinct metrics, here is why :
If you are an administrator, you can see your total custom metrics per hour as well as the top 500 custom metrics by cardinality in your account in the usage details page. You can also see this metric count on your metric summary page, where you’d see, clicking on the service.request.count metric, the exact number of unique tag combinations:
So if you only had the first host from the example above reporting, you’d have this:
Adding the second host:
Adding the third host as per the table above, you get your 13 distinct metrics:
Using the query editor, you can also find this using the count: aggregator
Ultimately, you’ll have 13 metrics using the following query: