Network Performance Monitoring is now generally available! Network Monitoring is now available!

Sending large volumes of metrics

DogStatsD works by sending metrics generated from your application to the Agent over a transport protocol. This protocol can be UDP (User Datagram Protocol) or UDS (Unix Domain Socket).

When DogStatsD is used to send a large volume of metrics to a single Agent, if proper measures are not taken, it is common to end up with the following symptoms:

  • High Agent CPU usage
  • Dropped datagrams / metrics
  • The DogStatsD client library (UDS) returning errors

Most of the time the symptoms can be alleviated by tweaking some configuration options described below.

General tips

Enable buffering on your client

Some StatsD and DogStatsD clients, by default, send one metric per datagram. This adds considerable overhead on the client, the operating system, and the Agent. If your client supports buffering multiple metrics in one datagram, enabling this option brings noticeable improvements.

Here are a few examples for official DogStatsD supported clients:

By using Datadog’s official Golang library datadog-go, the example below creates a buffered DogStatsD client instance with 256 maximum buffered metrics which means that all metrics sent from this instance of the client are buffered and sent in packets containing a maximum of 256 metrics:

package main

import (
	"log"
	"github.com/DataDog/datadog-go/statsd"
)

func main() {

  statsd, err := statsd.New("127.0.0.1:8125",
                 statsd.Buffered(),
                 statsd.WithMaxMessagesPerPayload(256),
                )
	if err != nil {
    		log.Fatal(err)
	}

  statsd.Gauge("example_metric.gauge", 1, []string{"env:dev"}, 1)
}

By using Datadog’s official Python library datadogpy, the example below creates a buffered DogStatsD client instance that sends up to 25 metrics in one packet when the block completes:

from datadog import DogStatsd

with DogStatsd(host="127.0.0.1", port=8125, max_buffer_size=25) as batch:
    batch.gauge('example_metric.gauge_1', 123, tags=["environment:dev"])
    batch.gauge('example_metric.gauge_2', 1001, tags=["environment:dev"])

By using Datadog’s official Ruby library dogstatsd-ruby, the example below creates a buffered DogStatsD client instance that sends metrics in one packet when the block completes:

require 'datadog/statsd'

statsd = Datadog::Statsd.new('127.0.0.1', 8125)

statsd.batch do |s|
  s.increment('example_metric.increment', tags: ['environment:dev'])
  s.gauge('example_metric.gauge', 123, tags: ['environment:dev'])
end

By using Datadog’s official Java library java-dogstatsd-client, the example below creates a buffered DogStatsD client instance with 256 maximum buffered metrics which means that all metrics sent from this instance of the client are buffered and sent in packets containing a maximum of 256 metrics:

import com.timgroup.statsd.NonBlockingStatsDClient;
import com.timgroup.statsd.StatsDClient;
import java.util.Random;

public class DogStatsdClient {

    public static void main(String[] args) throws Exception {

        StatsDClient Statsd = new NonBlockingStatsDClient("namespace", "127.0.0.1", 8125, 256);

        Statsd.incrementCounter("example_metric.increment", ["environment:dev"]);
        Statsd.recordGaugeValue("example_metric.gauge", 100, ["environment:dev"]);
    }
}

By using Datadog’s official C# library dogstatsd-csharp-client, the example below creates a DogStatsD client with UDP as transport:

using StatsdClient;

public class DogStatsdClient
{
    public static void Main()
    {
      StatsdUDP udp = new StatsdUDP("127.0.0.1", 8125);

      // Create a stats instance with "udp" as transport
      Statsd s = new Statsd(udp);
      s.Add<Statsd.Counting,int>("example_metric.count", 1, tags: new[] {"environment:dev"});
      s.Add("event title", "content", priority: "low");
      s.Add<Statsd.Counting,int>("example_metric.count", 1, tags: new[] {"environment:dev"});

      // All metrics buffered before this call will be sent in one packet
      s.Send();
    }
}

By using Datadog’s official PHP library php-datadogstatsd, the example below creates a buffered DogStatsD client instance that sends metrics in one packet when the block completes:

<?php

require __DIR__ . '/vendor/autoload.php';

  use DataDog\BatchedDogStatsd;

$client = new BatchedDogStatsd(
  array('host' => '127.0.0.1',
          'port' => 8125,
     )
);

$client->increment('example_metric.increment', array('environment'=>'dev'));
$client->increment('example_metric.increment', $sampleRate->0.5 , array('environment'=>'dev'));

Sample your metrics

It is possible to reduce the traffic from your DogStatsD client to the Agent by setting a sample rate value for your client. For example, a sample rate of 0.5 halves the number of UDP packets sent. This solution is a trade-off: you decrease traffic but slightly lose in precision and granularity.

For more information and code examples, see DogStatsD “Sample Rate” Parameter Explained.

Use DogStatsD over UDS (Unix Domain Socket)

UDS is an inter-process communication protocol used to transport DogStatsD payloads. It has very little overhead when compared to UDP and lowers the general footprint of DogStatsD on your system.

Operating System kernel buffers

Most operating systems add incoming UDP and UDS datagrams containing your metrics to a buffer with a maximum size. Once the max is reached, datagrams containing your metrics start getting dropped. It is possible to adjust the values to give the Agent more time to process incoming metrics:

Over UDP (User Datagram Protocol)

Linux

On most Linux distributions, the maximum size of the kernel buffer is set to 212992 by default (208 KiB). This can be confirmed using the following commands:

$ sysctl net.core.rmem_max
net.core.rmem_max = 212992

To set the maximum size of the DogStatsD socket buffer to 25MiB run:

$ sysctl -w net.core.rmem_max=26214400

Add the following configuration to /etc/sysctl.conf to make this change permanent:

net.core.rmem_max = 26214400

Then set the Agent dogstatsd_so_rcvbuf configuration option to the same number in datadog.yaml:

dogstatsd_so_rcvbuf: 26214400

Client side telemetry

Dogstatsd clients send telemetry metrics by default to the Agent. This allows you to better troubleshoot where bottleneck exists. Each metric will be tagged with the client language and the client version. These metrics will not be counted as custom metrics and will not be billed.

Each client shares a set of common tags.

TagDescriptionExample
clientThe language of the clientclient:py
client_versionThe version of the clientclient_version:1.2.3
client_transportThe transport byte the client (udp or uds)client_transport:uds

Note: When using UDP, network errors can’t be detected by the client and the corresponding metrics will not reflect bytes/packets drop.

Starting with version 0.34.0 of the Python client.

Metrics NameMetric TypeDescription
datadog.dogstatsd.client.metricscountNumber of metrics sent to the DogStatsD client by your application (before sampling).
datadog.dogstatsd.client.eventscountNumber of events sent to the DogStatsD client by your application.
datadog.dogstatsd.client.service_checkscountNumber of service_checks sent to the DogStatsD client by your application.
datadog.dogstatsd.client.bytes_sentcountNumber of bytes successfully sent to the Agent.
datadog.dogstatsd.client.bytes_droppedcountNumber of bytes dropped by the DogStatsD client.
datadog.dogstatsd.client.packets_sentcountNumber of datagrams successfully sent to the Agent.
datadog.dogstatsd.client.packets_droppedcountNumber of datagrams dropped by the DogStatsD client.

To disable telemetry, use the disable_telemetry method:

statsd.disable_telemetry()

See DataDog/datadogpy for more information about the client configuration.

Starting with version 4.6.0 of the Ruby client.

Metrics NameMetric TypeDescription
datadog.dogstatsd.client.metricscountNumber of metrics sent to the DogStatsD client by your application (before sampling).
datadog.dogstatsd.client.eventscountNumber of events sent to the DogStatsD client by your application.
datadog.dogstatsd.client.service_checkscountNumber of service_checks sent to the DogStatsD client by your application.
datadog.dogstatsd.client.bytes_sentcountNumber of bytes successfully sent to the Agent.
datadog.dogstatsd.client.bytes_droppedcountNumber of bytes dropped by the DogStatsD client.
datadog.dogstatsd.client.packets_sentcountNumber of datagrams successfully sent to the Agent.
datadog.dogstatsd.client.packets_droppedcountNumber of datagrams dropped by the DogStatsD client.

To disable telemetry, set the disable_telemetry parameter to true:

Datadog::Statsd.new('localhost', 8125, disable_telemetry: true)

See DataDog/dogstatsd-ruby for more information about the client configuration.

Starting with version 3.4.0 of the Go client.

Metric nameMetric TypeDescription
datadog.dogstatsd.client.metricscountNumber of metrics sent to the DogStatsD client by your application (before sampling).
datadog.dogstatsd.client.eventscountNumber of events sent to the DogStatsD client by your application.
datadog.dogstatsd.client.service_checkscountNumber of service_checks sent to the DogStatsD client by your application.
datadog.dogstatsd.client.bytes_sentcountNumber of bytes successfully sent to the Agent.
datadog.dogstatsd.client.bytes_droppedcountNumber of bytes dropped by the DogStatsD client.
datadog.dogstatsd.client.bytes_dropped_queuecountNumber of bytes dropped because the DogStatsD client queue was full.
datadog.dogstatsd.client.bytes_dropped_writercountNumber of bytes dropped because of an error while writing to Datadog.
datadog.dogstatsd.client.packets_sentcountNumber of datagrams successfully sent to the Agent.
datadog.dogstatsd.client.packets_droppedcountNumber of datagrams dropped by the DogStatsD client.
datadog.dogstatsd.client.packets_dropped_queuecountNumber of datagrams dropped because the DogStatsD client queue was full.
datadog.dogstatsd.client.packets_dropped_writercountNumber of datagrams dropped because of an error while writing to Datadog.

To disable telemetry, use the WithoutTelemetry setting:

statsd, err: = statsd.New("127.0.0.1:8125", statsd.WithoutTelemetry())

See DataDog/datadog-go for more information about the client configuration.

Telemetry will soon be added to the Java client.

Telemetry will soon be added to the PHP client.

Telemetry will soon be added to the .NET client.

Further Reading