New announcements for Serverless, Network, RUM, and more from Dash! New announcements from Dash!

Sending Custom Metrics to DogStatsD for Kubernetes

To emit custom metrics from your Kubernetes application, use DogStatsD, a metrics aggregation service bundled with the Datadog Agent. DogStatsD implements the StatsD protocol with some differences. See more on the DogStatsD documentation.

Use DogStatsD over a Unix Domain Socket

You can use DogStatsD over a Unix Domain Socket.

Create a listening socket

Edit your datadog.yaml file to set the dogstatsd_socket option to the path where DogStatsD should create its listening socket:

dogstatsd_socket: /var/run/datadog/dsd.socket

And restart your Agent. You can also set the socket path via the DD_DOGSTATSD_SOCKET environment variable.

Share the socket path with your application

The socket file needs to be accessible to the client containers. Mount a host directory on both sides: read-only in client containers, and read-write in the Agent container.

In your Agent container:

volumeMounts:
  - name: dsdsocket
    mountPath: /var/run/datadog
...
volumes:
- hostPath:
    path: /var/run/datadog/
  name: dsdsocket

In your client containers:

volumeMounts:
  - name: dsdsocket
    mountPath: /var/run/datadog
    readOnly: true
...
volumes:
- hostPath:
    path: /var/run/datadog/
  name: dsdsocket

For more details, see the DogStatsD over a Unix Domain Socket documentation.

Alternatively, use hostPort

Bind the DogStatsD port to a host port

  1. Add a hostPort to your datadog-agent.yaml manifest:

      ports:
        - containerPort: 8125
          hostPort: 8125
          name: dogstatsdport
          protocol: UDP

    This enables your applications to send metrics via DogStatsD on port 8125 on whichever node they happen to be running.

    Note: hostPort functionality requires a networking provider that adheres to the CNI specification, such as Calico, Canal, or Flannel. For more information, including a workaround for non-CNI network providers, consult the Kubernetes documentation.

  2. Enable DogStatsD non local traffic to allow StatsD data collection, set DD_DOGSTATSD_NON_LOCAL_TRAFFIC to true in your datadog-agent.yaml manifest:

    - name: DD_DOGSTATSD_NON_LOCAL_TRAFFIC
      value: "true"
    

    This allows collecting StatsD data from other containers than the one running the Agent.

  3. Apply the change:

      kubectl apply -f datadog-agent.yaml
    

Warning: The hostPort parameter opens a port on your host. Make sure your firewall only allows access from your applications or trusted sources. Another word of caution: some network plugins don’t support hostPorts yet, so this won’t work. The workaround in this case is to add hostNetwork: true in your Agent pod specifications. This shares the network namespace of your host with the Datadog Agent. It also means that all ports opened on the container are also opened on the host. If a port is used both on the host and in your container, they conflict (since they share the same network namespace) and the pod will not start. Not all Kubernetes installations allow this.

Pass the node’s IP address to your application

Your application needs a reliable way to determine the IP address of its host. This is made simple in Kubernetes 1.7, which expands the set of attributes you can pass to your pods as environment variables. In versions 1.7 and above, you can pass the host IP to any pod by adding an environment variable to the PodSpec. For instance, your application manifest might look like this:

env:
  - name: DOGSTATSD_HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

With this, any pod running your application is able to send DogStatsD metrics via port 8125 on $DOGSTATSD_HOST_IP.

Origin detection over UDP

Origin detection allows DogStatsD to detect where the container metrics come from, and tag metrics automatically. When this mode is enabled, all metrics received via UDP are tagged by the same container tags as Autodiscovery metrics.

Note: An alternative to UDP is Unix Domain Sockets.

To enable origin detection over UDP, add the following lines to your application manifest:

env:
  - name: DD_ENTITY_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid

Origin detection is supported in Agent 6.10.0+ and in the following client libraries:

LibraryVersion
Go2.2
PHP1.4.0
Python0.28.0
Ruby4.2.0
C#3.3.0
Java2.6

To set tag cardinality for the metrics collected using origin detection, use the environment variable DD_DOGSTATSD_TAG_CARDINALITY.

There are two environment variables that set tag cardinality: DD_CHECKS_TAG_CARDINALITY and DD_DOGSTATSD_TAG_CARDINALITY—as DogStatsD is priced differently, its tag cardinality setting is separated in order to provide the opportunity for finer configuration. Otherwise, these variables function the same way: they can have values low, orchestrator, or high. They both default to low.

Instrument your code to send metrics to DogStatsD

Once your application can send metrics via DogStatsD on each node, you can instrument your application code to submit custom metrics.

See the full list of Datadog DogStatsD Client Libraries

For instance, if your application is written in Go, import Datadog’s Go library, which provides a DogStatsD client library:

import "github.com/DataDog/datadog-go/statsd"

Before you can add custom counters, gauges, etc., initialize the StatsD client with the location of the DogStatsD service, depending on the method you have chosen:

  • Unix Domain Socket: $DD_DOGSTATSD_SOCKET
  • hostPort: $DOGSTATSD_HOST_IP
func main(){

  // other main() code omitted for brevity

  // use host IP and port to define endpoint
  dogstatsd, err := statsd.New(os.Getenv("DOGSTATSD_HOST_IP") + ":8125")
  // alternatively, use the unix socket path
  // dogstatsd, err = statsd.New(os.Getenv("DD_DOGSTATSD_SOCKET"))
  if err != nil{
    log.Printf("Cannot get a DogStatsD client.")
  } else {
    // prefix every metric and event with the app name
    dogstatsd.Namespace = "My Application"

    // post an event to Datadog at app startup
    dogstatsd.Event(&statsd.Event{
      Title: "My application started.",
      Text: "My application started.",
      })
  }
}

You can also increment a custom metric for each of your handler functions. For example, every time the InfoHandler function is called, it increments the request_count metric by 1, while applying the tag endpoint:info to that data point:

func InfoHandler(rw http.ResponseWriter, req *http.Request) {
    dogstatsd.Incr("request_count", []string{"endpoint:info"}, 1)
    info := HandleError(masterPool.Get(0).Do("INFO")).([]byte)
    rw.Write(info)
}

Further Reading

Additional helpful documentation, links, and articles: