DogStatsD

The easiest way to get your custom application metrics into Datadog is to send them to DogStatsD, a metrics aggregation service bundled with the Datadog Agent. DogStatsD implements the StatsD protocol and adds a few Datadog-specific extensions:

  • Histogram metric type
  • Service checks
  • Events
  • Tagging

Any compliant StatsD client works with DogStatsD and the Agent, but does not include the Datadog-specific extensions.

Note: DogStatsD does NOT implement timers from StatsD as a native metric type (though it does support them through histograms).

DogStatsD is available on Docker Hub and GCR:

Docker HubGCR
hub.docker.com/r/datadog/dogstatsdgcr.io/datadoghq/dogstatsd

How it works

DogStatsD accepts custom metrics, events, and service checks over UDP and periodically aggregates and forwards them to Datadog.

Because it uses UDP, your application can send metrics to DogStatsD and resume its work without waiting for a response. If DogStatsD ever becomes unavailable, your application doesn’t experience an interruption.

dogstatsd

As it receives data, DogStatsD aggregates multiple data points for each unique metric into a single data point over a period of time called the flush interval. DogStatsD uses a flush interval of 10 seconds.

Setup

DogStatsD is enabled by default over UDP port 8125 for Agent v6+. If you don’t need to change this port, see directly how to setup DogStatsD in your code.

Agent

    By default, DogStatsD listens on UDP port 8125. If you need to change this, configure the dogstatsd_port option in the main Agent configuration file, and restart the Agent. You can also configure DogStatsD to use a Unix domain socket. To enable a custom Agent DogStatsD server UDP port:

    1. Edit your datadog.yaml file to un-comment the use_dogstatsd and dogstatsd_port parameters:

      ## @param use_dogstatsd - boolean - optional - default: true
## Set this option to false to disable the Agent DogStatsD server.
#
use_dogstatsd: true

## @param dogstatsd_port - integer - optional - default: 8125
## Override the Agent DogStatsD port.
## Note: Make sure your client is sending to the same UDP port.
#
dogstatsd_port: 8125

    2. Restart your Agent.

    By default, DogStatsD listens on UDP port 8125, so you need to bind this port to your host port when running the Agent in a container. If your StatsD metrics come from outside of localhostyou must set DD_DOGSTATSD_NON_LOCAL_TRAFFIC to true to allow metric collection. In order to run the Agent with the DogStatsd server up, execute the following command:

    docker run -d --cgroupns host \
              --pid host \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -e DD_API_KEY=<DATADOG_API_KEY> \
              -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC="true" \
              -p 8125:8125/udp \
              gcr.io/datadoghq/agent:latest

    If you need to change the port used to collect StatsD metrics, use the DD_DOGSTATSD_PORT="<NEW_DOGSTATSD_PORT> environment variable. You can also configure DogStatsD to use a Unix domain socket:

    To start collecting your StatsD metrics, you need to bind the DogStatsD port to a host port. You can also configure DogStatsD to use a Unix domain socket.

    1. Add a hostPort to your datadog-agent.yaml manifest:

      ports:
    - containerPort: 8125
      hostPort: 8125
      name: dogstatsdport
      protocol: UDP

      This enables your applications to send metrics with DogStatsD on port 8125 on whichever node they happen to be running.

      Note: hostPort functionality requires a networking provider that adheres to the CNI specification, such as Calico, Canal, or Flannel. For more information, including a workaround for non-CNI network providers, see the Kubernetes documentation: HostPort services do not work.

      Note: For an Operator deployment, configure the host port with agent.config.hostPort.

    2. Enable DogStatsD non local traffic to allow StatsD data collection, set DD_DOGSTATSD_NON_LOCAL_TRAFFIC to true in your datadog-agent.yaml manifest:

      - name: DD_DOGSTATSD_NON_LOCAL_TRAFFIC
  value: 'true'

      This allows collecting StatsD data from other containers than the one running the Agent.

    3. Apply the change:

      kubectl apply -f datadog-agent.yaml

    Warning: The hostPort parameter opens a port on your host. Make sure your firewall only allows access from your applications or trusted sources. If your network plugin doesn’t support hostPorts, so add hostNetwork: true in your Agent pod specifications. This shares the network namespace of your host with the Datadog Agent. It also means that all ports opened on the container are opened on the host. If a port is used both on the host and in your container, they conflict (since they share the same network namespace) and the pod does not start. Some Kubernetes installations do not allow this.

    Send StatsD metrics to the Agent

    Your application needs a reliable way to determine the IP address of its host. This is made simple in Kubernetes 1.7, which expands the set of attributes you can pass to your pods as environment variables. In versions 1.7 and above, you can pass the host IP to any pod by adding an environment variable to the PodSpec. For instance, your application manifest might look like this:

    env:
    - name: DD_AGENT_HOST
      valueFrom:
          fieldRef:
              fieldPath: status.hostIP

    With this, any pod running your application is able to send DogStatsD metrics with port 8125 on $DD_AGENT_HOST.

    Note: As a best practice, Datadog recommends using unified service tagging when assigning attributes. Unified service tagging ties Datadog telemetry together through the use of three standard tags: env, service, and version. To learn how to unify your environment, see unified service tagging.

    Origin detection over UDP

    Origin detection is supported in Agent 6.10.0+ and allows DogStatsD to detect where the container metrics come from, and tag metrics automatically. When this mode is enabled, all metrics received through UDP are tagged by the same pod tags as Autodiscovery metrics.

    Notes:

    • Origin detection with UDP uses the pod ID as the entity ID, so container-level tags are not emitted.
    • An alternative to UDP is Unix Domain Sockets.

    To enable origin detection over UDP, add the following lines to your application manifest:

    env:
    - name: DD_ENTITY_ID
      valueFrom:
          fieldRef:
              fieldPath: metadata.uid

    To set tag cardinality for the metrics collected using origin detection, set the environment variable DD_DOGSTATSD_TAG_CARDINALITY to either low (default) or orchestrator.

    Note: For UDP, pod_name tags are not added by default to avoid creating too many custom metrics.

    To gather custom metrics with DogStatsD with helm:

    1. Update your datadog-values.yaml file to enable DogStatsD:

        dogstatsd:
    port: 8125
    useHostPort: true
    nonLocalTraffic: true

      Note: hostPort functionality requires a networking provider that adheres to the CNI specification, such as Calico, Canal, or Flannel. For more information, including a workaround for non-CNI network providers, see the Kubernetes documentation: HostPort services do not work.

      Warning: The hostPort parameter opens a port on your host. Make sure your firewall only allows access from your applications or trusted sources. If your network plugin doesn’t support hostPorts, so add hostNetwork: true in your Agent pod specifications. This shares the network namespace of your host with the Datadog Agent. It also means that all ports opened on the container are opened on the host. If a port is used both on the host and in your container, they conflict (since they share the same network namespace) and the pod does not start. Some Kubernetes installations do not allow this.

    2. Upgrade your Agent configuration:

      helm upgrade -f datadog-values.yaml <RELEASE_NAME> datadog/datadog

    3. Update your application pods: Your application needs a reliable way to determine the IP address of its host. This is made simple in Kubernetes 1.7, which expands the set of attributes you can pass to your pods as environment variables. In versions 1.7 and above, you can pass the host IP to any pod by adding an environment variable to the PodSpec. For instance, your application manifest might look like this:

      env:
    - name: DD_AGENT_HOST
      valueFrom:
          fieldRef:
              fieldPath: status.hostIP

      With this, any pod running your application is able to send DogStatsD metrics through port 8125 on $DD_AGENT_HOST.

    Code

    Install the DogStatsD client

    Official Datadog-DogStatsD client libraries are available for the following languages. Any compliant StatsD client works with DogStatsD and the Agent, but does not include the Datadog-specific features mentioned above:

    pip install datadog

    gem install dogstatsd-ruby

    go get github.com/DataDog/datadog-go/v5/statsd

    The Java DataDog StatsD Client is distributed with maven central, and can be downloaded from Maven. Start by adding the following configuration to your pom.xml:

    <dependency>
    <groupId>com.datadoghq</groupId>
    <artifactId>java-dogstatsd-client</artifactId>
    <version>3.0.0</version>
</dependency>

    Add the following to your composer.json:

    "datadog/php-datadogstatsd": "1.4.*"

    Note: The first version shipped in Composer is 0.0.3

    Or manually clone the repository at github.com/DataDog/php-datadogstatsd and set it up with require './src/DogStatsd.php'.

    Instantiate the DogStatsD client

    Once your DogStatsD client is installed, instantiate it in your code:

    from datadog import initialize, statsd

options = {
    'statsd_host':'127.0.0.1',
    'statsd_port':8125
}

initialize(**options)
    By default, Python DogStatsD client instances (including the statsd global instance) cannot be shared across processes but are thread-safe. Because of this, the parent process and each child process must create their own instances of the client or the buffering must be explicitly disabled by setting disable_buffering to True. See the documentation on datadog.dogstatsd for more details.
    # Import the library
require 'datadog/statsd'

# Create a DogStatsD client instance.
statsd = Datadog::Statsd.new('localhost', 8125)

    dogstatsd_client, err := statsd.New("127.0.0.1:8125")
if err != nil {
    log.Fatal(err)
}

    For more options, see Datadog’s GoDoc.

    import com.timgroup.statsd.NonBlockingStatsDClientBuilder;
import com.timgroup.statsd.StatsDClient;

public class DogStatsdClient {

    public static void main(String[] args) throws Exception {

        StatsDClient statsd = new NonBlockingStatsDClientBuilder()
            .prefix("statsd")
            .hostname("localhost")
            .port(8125)
            .build();


        // alternatively
        StatsDClient statsdAlt = new NonBlockingStatsDClient(
            new NonBlockingStatsDClientBuilder(
                .prefix("statsd")
                .hostname("localhost")
                .port(8125)
                .resolve()));

    }
}

    Instantiate a new DogStatsd object using composer:

    <?php

require __DIR__ . '/vendor/autoload.php';

use DataDog\DogStatsd;

$statsd = new DogStatsd(
    array('host' => '127.0.0.1',
          'port' => 8125,
     )
  );

    Configure the DogStatsd class:

    // The code is located under the StatsdClient namespace
using StatsdClient;

// ...

var dogstatsdConfig = new StatsdConfig
{
    StatsdServerName = "127.0.0.1",
    StatsdPort = 8125,
};

using (var dogStatsdService = new DogStatsdService())
{
    if (!dogStatsdService.Configure(dogstatsdConfig))
        throw new InvalidOperationException("Cannot initialize DogstatsD. Set optionalExceptionHandler argument in the `Configure` method for more information.");
    // ...
} // Flush metrics not yet sent

    If you use DogStatsD with the Container Agent or in Kubernetes, you must instantiate the host to which StatsD metrics are forwarded to with the $DD_DOGSTATSD_SOCKET environment variable if using a Unix Domain Socket, or with the $DD_AGENT_HOST environment variable if you are using the host port binding method.

    Client instantiation parameters

    Note: As a best practice, Datadog recommends using unified service tagging when assigning tags. Unified service tagging ties Datadog telemetry together through the use of three standard tags: env, service, and version. To learn how to unify your environment, see unified service tagging.

    In addition to the required DogStatsD configuration (url and port), the following optional parameters are available for your DogStatsD client:

    ParameterTypeDefaultDescription
    statsd_hostStringlocalhostThe host of your DogStatsD server.
    statsd_portInteger8125The port of your DogStatsD server.
    statsd_socket_pathStringnullThe path to the DogStatsD Unix domain socket (overrides host and port, only supported with the Agent v6+).
    statsd_constant_tagsList of stringsnullTags to apply to all metrics, events, and service checks.
    statsd_namespaceStringnullNamespace to prefix all metrics, events, and service checks.

    For the full list of optional parameters available for datadog.initialize() as well as parameters only available when explicitly instantiating datadog.dogstatsd.DogStatsd instances, see the Datadog Python library.

    ParameterTypeDefaultDescription
    hostStringlocalhostThe host of your DogStatsD server.
    portInteger8125The port of your DogStatsD server.
    socket_pathStringnullThe path to the DogStatsD Unix domain socket (overrides host and port, only supported with the Agent v6+).
    tagsList of stringsnullTags to apply to all metrics, events, and service checks.
    namespaceStringnullNamespace to prefix to all metrics, events, and service checks.
    single_threadBooleanfalseMakes the client send the metrics on the main thread when enabled rather than in a companion thread.

    For the full list of optional parameters, see the dogstatsd-ruby repo on GitHub.

    The Go client has multiple options for configuring the behavior of your client.

    ParameterTypeDescription
    WithNamespace()StringConfigure a namespace to prefix to all metrics, events, and service checks.
    WithTags()List of stringsGlobal tags applied to every metric, event, and service check.

    For all available options, see Datadog’s GoDoc.

    As of v2.10.0 the recommended way to instantiate the client is with the NonBlockingStatsDClientBuilder. You can use the following builder methods to define the client parameters.

    Builder MethodTypeDefaultDescription
    prefix(String val)StringnullThe prefix to apply to all metrics, events, and service checks.
    hostname(String val)StringlocalhostThe host name of the targeted StatsD server.
    port(int val)Integer8125The port of the targeted StatsD server.
    constantTags(String... val)String varargsnullGlobal tags to be applied to every metric, event, and service check.
    blocking(boolean val)BooleanfalseThe type of client to instantiate: blocking vs non-blocking.
    socketBufferSize(int val)Integer-1The size of the underlying socket buffer.
    enableTelemetry(boolean val)BooleanfalseClient telemetry reporting.
    entityID(String val)StringnullEntity ID for origin detection.
    errorHandler(StatsDClientErrorHandler val)IntegernullError handler in case of an internal client error.
    maxPacketSizeBytes(int val)Integer8192/1432The maximum packet size; 8192 over UDS, 1432 for UDP.
    processorWorkers(int val)Integer1The number of processor worker threads assembling buffers for submission.
    senderWorkers(int val)Integer1The number of sender worker threads submitting buffers to the socket.
    poolSize(int val)Integer512Network packet buffer pool size.
    queueSize(int val)Integer4096Maximum number of unprocessed messages in the queue.
    timeout(int val)Integer100the timeout in milliseconds for blocking operations. Applies to unix sockets only.

    For more information, search the Java DogStatsD package for the NonBlockingStatsDClient Class and NonBlockingStatsDClientBuilder Class. Make sure you view the version that matches your client release.

    ParameterTypeDefaultDescription
    hostStringlocalhostThe host of your DogStatsD server. If this is not set the Agent looks at the DD_AGENT_HOST environment variable.
    portInteger8125The port of your DogStatsD server. If this is not set, the Agent looks at the DD_DOGSTATSD_PORT environment variable.
    socket_pathStringnullThe path to the DogStatsD Unix domain socket (overrides host and port). This is only supported with Agent v6+.
    global_tagsList of StringsnullTags to apply to all metrics, events, and service checks. The @dd.internal.entity_id tag is appended to global_tags from the DD_ENTITY_ID environment variable.
    ParameterTypeDefaultDescription
    StatsdServerNameStringlocalhostThe host name of the targeted StatsD server.
    StatsdPortInteger8125The port of the targeted StatsD server.
    PrefixStringnullPrefix to apply to every metric, event, and service check.
    ConstantTagsList of stringsnullGlobal tags to be applied to every metric, event, and service check.

    Dive into DogStatsD

    DogStatsD and StatsD are broadly similar, however, DogStatsD contains advanced features which are specific to Datadog, including available data types, events, service checks, and tags:

    Additional helpful documentation, links, and articles:


    If you’re interested in learning more about the datagram format used by DogStatsD, or want to develop your own Datadog library, see the datagram and shell usage section, which also explains how to send metrics and events straight from the command line.