Sending large volumes of metrics

DogStatsD works by sending metrics generated from your application to the Agent over a transport protocol. This transport protocol can be either UDP (User Datagram Protocol) or UDS (Unix Domain Socket).

When DogStatsD is used to send a large volume of metrics to a single Agent, if proper measures are not taken, it is common to end up with the following symptoms:

• High agent CPU usage
• Dropped datagrams / metrics
• (UDS) DogStatsD client library returning lots of errors

Most of the time those symptoms can be alleviated by tweaking some configuration options described below.

General tips

Enable buffering on your client

Some StatsD and DogStatsD clients, by default, send one metric per datagram. This adds considerable overhead on the client, the operating system, and the Agent. If your client supports buffering multiple metrics in one datagram, enabling this option brings noticeable improvements.

Here are a few examples for supported clients:

// Create a buffered dogstatsd client instance with 256 maximum buffered metrics
client, err := statsd.New("127.0.0.1:8125",
    statsd.Buffered(), // enable buffering
    statsd.WithMaxMessagesPerPayload(256), // sets the maximum number of messages in a single datagram
)
// All metrics sent from this instance of the client will be buffered and sent in
// packets containing several metrics
client.Gauge("kafka.health", 1, []string{"env:production", "partition:1", "partition:2"}, 1)

# Load the dogstats module.
from datadog import DogStatsd

# All metrics will be buffered and sent in one packet when the block completes
with DogStatsd(host="127.0.0.1", port=8125, max_buffer_size=25) as batch:
    batch.gauge('users.online', 123)
    batch.gauge('active.connections', 1001)

require 'datadog/statsd'

# Create a stats instance
statsd = Datadog::Statsd.new('127.0.0.1', 8125)

# All metrics will be buffered and sent in one packet when the block completes
statsd.batch do |s|
  s.increment('page.views')
  s.gauge('users.online', 123)
end

// Create a stats instance with a buffer size of 256
private static final StatsDClient statsd = new NonBlockingStatsDClient("namespace", "127.0.0.1", 8125, 256);

// All metrics sent from this instance of the client will be buffered and sent in
// packets containing several metrics
public void foo() {
    statsd.incrementCounter("foo");
    statsd.recordGaugeValue("bar", 100);
}

// Create the transport
StatsdUDP udp = new StatsdUDP("127.0.0.1", 8125);

using (udp)
{
    // Create a stats instance with "udp" as transport
    Statsd s = new Statsd(udp);

    s.Add<Statsd.Counting,int>("stat-name", 1);
    s.Add("event title", "content", priority: "low");
    s.Add<Statsd.Counting,int>("stat-name", 1);

    // All metrics buffered before this call will be sent in one packet
    s.Send();
}

require __DIR__ . '/vendor/autoload.php';

use DataDog\BatchedDogStatsd;

$client = new BatchedDogStatsd();

// All metrics sent from this instance of the client will be buffered and sent in
// packets containing several metrics
$client->increment('your.data.point');
$client->increment('your.data.point', .5);

Please refer to your client documentation for additional details.

Sample your metrics

It is possible to reduce the traffic from your DogStatsD client to the Agent by setting a sample rate value for your client. For example, a sample rate of 0.5 halves the number of UDP packets sent. This solution is a trade-off: you decrease traffic but slightly lose in precision and granularity.

For more information, and code examples. see DogStatsD “Sample Rate” Parameter Explained.

Use DogStatsD over UDS (Unix Domain Socket)

UDS is an inter-process communication protocol used to transport DogStatsD payloads. It has very little overhead when compared to UDP and lowers the general footprint of DogStatsD on your system.

Operating System kernel buffers

Most OSs add incoming UDP and UDS datagrams containing your metrics to a buffer with a maximum size. Once this size is reached, datagrams containing your metrics starts getting dropped.

It is possible to adjust those values to give the Agent more time to process incoming metrics.

Over UDP (User Datagram Protocol)

Linux

On most Linux distributions, the maximum size of the kernel buffer is set to 212992 by default (208 KiB). This can be confirmed using the following commands:

$ sysctl net.core.rmem_max
net.core.rmem_max = 212992

To set the maximum size of the DogStatsD socket buffer to 25MiB run:

$ sysctl -w net.core.rmem_max=26214400

Add the following configuration to /etc/sysctl.conf to make this change permanent:

net.core.rmem_max = 26214400

Then set the Agent dogstatsd_so_rcvbuf configuration option to the same number in datadog.yaml:

dogstatsd_so_rcvbuf: 26214400