The Service Map for APM is here!

Sending Custom Metrics to DogStatsD for Kubernetes

To emit custom metrics from your Kubernetes application, use DogStatsD, a metrics aggregation service bundled with the Datadog Agent. DogStatsD implements the StatsD protocol with some differences. See more on the DogStatsD documentation.

Use DogStatsD over a Unix Domain Socket

You can use DogStatsD over a Unix Domain Socket.

Create a listening socket

Edit your datadog.yaml file to set the dogstatsd_socket option to the path where DogStatsD should create its listening socket:

dogstatsd_socket: /var/run/datadog/dsd.socket

And restart your Agent. You can also set the socket path via the DD_DOGSTATSD_SOCKET environment variable.

Share the socket path with your application

The socket file needs to be accessible to the client containers. Mount a host directory on both sides: read-only in client containers, and read-write in the Agent container.

In your Agent container:

volumeMounts:
  - name: dsdsocket
    mountPath: /var/run/datadog
...
volumes:
- hostPath:
    path: /var/run/datadog/
  name: dsdsocket

In your client containers:

volumeMounts:
  - name: dsdsocket
    mountPath: /var/run/datadog
    readOnly: true
...
volumes:
- hostPath:
    path: /var/run/datadog/
  name: dsdsocket

For more details, see the DogStatsD over a Unix Domain Socket documentation.

Alternatively, use hostPort

Bind the DogStatsD port to a host port

Add a hostPort to your datadog-agent.yaml file:

ports:
  - containerPort: 8125
    hostPort: 8125
    name: dogstatsdport
    protocol: UDP

This enables your applications to send metrics via DogStatsD on port 8125 on whichever node they happen to be running.

Note: hostPort functionality requires a networking provider that adheres to the CNI specification, such as Calico, Canal, or Flannel. For more information, including a workaround for non-CNI network providers, consult the Kubernetes documentation.

To apply the change:

kubectl apply -f datadog-agent.yaml

Pass the node’s IP address to your application

Your application needs a reliable way to determine the IP address of its host. This is made simple in Kubernetes 1.7, which expands the set of attributes you can pass to your pods as environment variables. In versions 1.7 and above, you can pass the host IP to any pod by adding an environment variable to the PodSpec. For instance, your application manifest might look like this:

env:
  - name: DOGSTATSD_HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

With this, any pod running your application is able to send DogStatsD metrics via port 8125 on $DOGSTATSD_HOST_IP.

Instrument your code to send metrics to DogStatsD

Once your application can send metrics via DogStatsD on each node, you can instrument your application code to submit custom metrics.

See the full list of Datadog DogStatsD Client Libraries

For instance, if your application is written in Go, import Datadog’s Go library, which provides a DogStatsD client library:

import "github.com/DataDog/datadog-go/statsd"

Before you can add custom counters, gauges, etc., initialize the StatsD client with the location of the DogStatsD service, depending on the method you have chosen:

  • Unix Domain Socket: $DD_DOGSTATSD_SOCKET
  • hostPort: $DOGSTATSD_HOST_IP
func main(){

  // other main() code omitted for brevity

  var err error
  // use host IP and port to define endpoint
  dogstatsd, err = statsd.New(os.Getenv("DOGSGTATSD_HOST_IP") + ":8125")
  // alternatively, use the unix socket path
  // dogstatsd, err = statsd.New(os.Getenv("DD_DOGSTATSD_SOCKET"))
  if err != nil{
    log.Printf("Cannot get a DogStatsD client.")
  } else {
    // prefix every metric and event with the app name
    dogstatsd.Namespace = "My Application"

    // post an event to Datadog at app startup
    dogstatsd.Event(*statsd.Event{
      Title: "My application started.",
      Text: "My application started.",
      })
  }
}

You can also increment a custom metric for each of your handler functions. For example, every time the InfoHandler function is called, it increments the request_count metric by 1, while applying the tag endpoint:info to that data point:

func InfoHandler(rw http.ResponseWriter, req *http.Request) {
    dogstatsd.Incr("request_count", []string{"endpoint:info"}, 1)
    info := HandleError(masterPool.Get(0).Do("INFO")).([]byte)
    rw.Write(info)
}

Further Reading

Additional helpful documentation, links, and articles: