Metric Submission: Custom Agent Check

Functions are used to submit metrics with a custom Agent check. Different functions are available depending on the metric type. Depending on the function used, the submission and actual metric type stored within Datadog might differ.

Functions

monotonic_count()

This function is used to track a raw COUNT metric that always increases. The Datadog Agent calculates the delta between each submission. Samples that have a lower value than the previous sample are ignored. Lower values usually indicate the underlying raw COUNT metric has been reset. The function can be called multiple times during a check’s execution.

For example, submitting samples 2, 3, 6, 7 sends a value of 5 (7-2) during the first check execution. Submitting the samples 10, 11 on the same monotonic_count sends a value of 4 (11-7) during the second check execution.

Note: Metrics submitted with this function are stored with a COUNT metric type in Datadog. Each value in the stored timeseries is a delta of the metric’s value between samples (not time-normalized).

Function template:

self.monotonic_count(name, value, tags=None, hostname=None, device_name=None)
ParameterTypeRequiredDefault ValueDescription
nameStringYes-The name of the metric.
valueFloatYes-The value for the metric.
tagsList of stringsNo-A list of tags to associate with this metric.
hostnameStringNoCurrent hostA hostname to associate with this metric.
device_nameStringNo-Deprecated. Adds a tag in the form device:<DEVICE_NAME> to the tags list instead.

count()

This function submits the number of events that occurred during the check interval. It can be called multiple times during a check’s execution, each sample being added to the value that is sent.

Note: Metrics submitted with this function are stored with a COUNT metric type in Datadog. Each value in the stored timeseries is a delta of the metric’s value between samples (not time-normalized).

Function template:

self.count(name, value, tags=None, hostname=None, device_name=None)
ParameterTypeRequiredDefault ValueDescription
nameStringYes-The name of the metric.
valueFloatYes-The value for the metric.
tagsList of stringsNo-A list of tags to associate with this metric.
hostnameStringNocurrent hostA hostname to associate with this metric.
device_nameStringNo-Deprecated. Adds a tag in the form device:<DEVICE_NAME> to the tags list instead.

gauge()

This function submits the value of a metric at a given timestamp. If called multiple times during a check’s execution for a metric only the last sample is used.

Note: Metrics submitted with this function are stored with a GAUGE metric type in Datadog.

Function template:

self.gauge(name, value, tags=None, hostname=None, device_name=None)
ParameterTypeRequiredDefault ValueDescription
nameStringYes-The name of the metric.
valueFloatYes-The value for the metric.
tagsList of stringsNo-A list of tags to associate with this metric.
hostnameStringNocurrent hostA hostname to associate with this metric.
device_nameStringNo-Deprecated. Adds a tag in the form device:<DEVICE_NAME> to the tags list instead.

rate()

This function submits the sampled raw value of your RATE metric. The Datadog Agent calculates the delta of that metric’s value between two submission, and divides it by the submission interval to get the rate. This function should only be called once during a check, otherwise it throws away any value that is less than a previously submitted value.

Note: Metrics submitted with this function are stored as a GAUGE metric type in Datadog. Each value in the stored timeseries is a time-normalized delta of the metric’s value between samples.

Function template:

self.rate(name, value, tags=None, hostname=None, device_name=None)
ParameterTypeRequiredDefault ValueDescription
nameStringYes-The name of the metric.
valueFloatYes-The value for the metric.
tagsList of stringsNo-A list of tags to associate with this metric.
hostnameStringNocurrent hostA hostname to associate with this metric.
device_nameStringNo-Deprecated. Adds a tag in the form device:<DEVICE_NAME> to the tags list instead.

histogram()

This function submits the sample of a histogram metric that occurred during the check interval. It can be called multiple times during a check’s execution. Each sample is added to the statistical distribution of the set of values for this metric.

Note: All metric aggregation produced are stored as a GAUGE metric type in Datadog, except the <METRIC_NAME>.count that is stored as a RATE metric type in Datadog.

Function template:

self.histogram(name, value, tags=None, hostname=None, device_name=None)
ParameterTypeRequiredDefault ValueDescription
nameStringYes-The name of the metric.
valueFloatYes-The value for the metric.
tagsList of stringsNo-A list of tags to associate with this metric.
hostnameStringNocurrent hostA hostname to associate with this metric.
device_nameStringNo-Deprecated. Adds a tag in the form device:<DEVICE_NAME> to the tags list instead.

Tutorial

Follow the steps below to create a custom Agent check that sends all metric types periodically:

  1. Create the directory metrics_example.d/ in the conf.d/ folder at the root of your Agent’s configuration directory.

  2. In metrics_example.d/ folder, create an empty configuration file named metrics_example.yaml with the following content:

    instances: [{}]
    
  3. Up one level from the conf.d/ folder, go to the checks.d/ folder. Create a custom check file named metrics_example.py with the content below:

    import random
    
    from datadog_checks.base import AgentCheck
    
    __version__ = "1.0.0"
    
    class MyClass(AgentCheck):
        def check(self, instance):
            self.count(
                "example_metric.count",
                2,
                tags=["env:dev","metric_submission_type:count"],
            )
            self.count(
                "example_metric.decrement",
                -1,
                tags=["env:dev","metric_submission_type:count"],
            )
            self.count(
                "example_metric.increment",
                1,
                tags=["env:dev","metric_submission_type:count"],
            )
            self.rate(
                "example_metric.rate",
                1,
                tags=["env:dev","metric_submission_type:rate"],
            )
            self.gauge(
                "example_metric.gauge",
                random.randint(0, 10),
                tags=["env:dev","metric_submission_type:gauge"],
            )
            self.monotonic_count(
                "example_metric.monotonic_count",
                2,
                tags=["env:dev","metric_submission_type:monotonic_count"],
            )
    
            # Calling the functions below twice simulates
            # several metrics submissions during one Agent run.
            self.histogram(
                "example_metric.histogram",
                random.randint(0, 10),
                tags=["env:dev","metric_submission_type:histogram"],
            )
            self.histogram(
                "example_metric.histogram",
                random.randint(0, 10),
                tags=["env:dev","metric_submission_type:histogram"],
            )
    
  4. Restart the Agent.

  5. Validate your custom check is running correctly with the Agent’s status subcommand. Look for metrics_example under the Checks section:

    =========
    Collector
    =========
    
      Running Checks
      ==============
    
        (...)
    
        metrics_example (1.0.0)
        -----------------------
          Instance ID: metrics_example:d884b5186b651429 [OK]
          Total Runs: 2
          Metric Samples: Last Run: 8, Total: 16
          Events: Last Run: 0, Total: 0
          Service Checks: Last Run: 0, Total: 0
          Average Execution Time : 2ms
    
        (...)
    
  6. Verify your metrics are reporting to Datadog on your Metric Summary page.

Further Reading

Additional helpful documentation, links, and articles: