Cloudfoundry

Agent Check

Overview

Any Cloud Foundry deployment can send metrics and events to Datadog. The data helps you track the health and availability of all nodes in the deployment, monitor the jobs they run, collect metrics from the Loggregator Firehose, and more. Use this page to learn how to monitor your application on Cloud Foundry and your Cloud Foundry cluster.

For Pivotal Cloud Foundry, you have the option to install the Datadog integration tiles with Ops Manager:

• Datadog Cluster Monitoring for PCF
• Datadog Application Monitoring for PCF

Monitor Your Applications on Cloud Foundry

Use Datadog Cloud Foundry Buildpack to monitor your Cloud Foundry application. This is a supply buildpack for Cloud Foundry that installs a Datadog DogStatsD binary and Datadog Agent in the container your app is running on.

Setup

Cloud Foundry < 1.12

Our buildpack uses Cloud Foundry multi-buildpack feature that was introduced in version 1.12.

For older versions, Cloud Foundry provides a back-port of this feature in the form of a buildpack. You must install and configure this backport in order to use Datadog’s buildpack:

1. Upload the multi-buildpack back-port. Download the latest multi-build pack release and upload it to your Cloud Foundry environment.

   cf create-buildpack multi-buildpack ./multi-buildpack-v-x.y.z.zip 99 --enable

2. Add a multi-buildpack manifest to your application. As detailed on the multi-buildpack back-port repo, create a multi-buildpack.yml file at the root of your application and configure it for your environment. Add a link to the Datadog Cloud Foundry Buildpack and to your regular buildpack:

     buildpacks:
       - "https://cloudfoundry.datadoghq.com/datadog-cloudfoundry-buildpack/datadog-cloudfoundry-buildpack-3.1.0.zip"
       - "https://github.com/cloudfoundry/ruby-buildpack#v1.7.18" # Replace this with your regular buildpack

   The URLs for the Datadog Buildpack are :

   • https://cloudfoundry.datadoghq.com/datadog-cloudfoundry-buildpack/datadog-cloudfoundry-buildpack-latest.zip
   • https://cloudfoundry.datadoghq.com/datadog-cloudfoundry-buildpack/datadog-cloudfoundry-buildpack-x.y.z.zip

   You should not use the latest here but instead pin it to a specific version (replace x.y.z by the specific version you want to use).

   Important: Your regular buildpack should probably be the last in the manifest to act as a final buildpack. To learn more refer to cloud foundry documentation about buildpacks.

3. Push your application with the multi-buildpack Ensure that the multi-buildpack is the buildpack selected by Cloud Foundry for your application :

   cf push <YOUR_APP> -b multi-buildpack

Cloud Foundry >= 1.12

1. Upload the Datadog Cloud Foundry Buildpack. Download the latest Datadog build pack release and upload it to your Cloud Foundry environment.

   cf create-buildpack datadog-cloudfoundry-buildpack ./datadog-cloudfoundry-buildpack-latest.zip

2. Push your application with the Datadog buildpack and your buildpacks. The process to push your application with multiple buildpack is described in the cloud foundry documentation.

   cf push <YOUR_APP> --no-start -b binary_buildpack
   cf v3-push <YOUR_APP> -b datadog-cloudfoundry-buildpack -b YOUR-BUILDPACK-1 -b YOUR-FINAL-BUILDPACK

Important: If you were using a single buildpack before, it should probably be the last one to be loaded in order for it to act as a final buildpack. To learn more refer to cloud foundry documentation about buildpacks.

Meta-Buildpack (deprecated)

If you are a meta-buildpack user, Datadog’s buildpack can be used as a decorator out of the box.

Note: The meta-buildpack has been deprecated by pivotal in favor of the multi-buildpack and that we might drop the support for it in a future release.

Configuration

Metric Collection

Set an API Key in your environment to enable the buildpack:

# set the environment variable
cf set-env <YOUR_APP> DD_API_KEY <DD_API_KEY>
# restage the application to get it to pick up the new environment variable and use the buildpack
cf restage <YOUR_APP>

Log Collection (closed beta)

To start collecting logs from your application in CloudFoundry, the Agent contained in the buildpack needs to be activated and log collection enabled.

cf set-env $YOUR_APP_NAME RUN_AGENT true
cf set-env $YOUR_APP_NAME DD_LOGS_ENABLED true
# Disable the Agent core checks to disable system metrics collection
cf set-env $YOUR_APP_NAME DD_ENABLE_CHECKS false
# restage the application to get it to pick up the new environment variable and use the buildpack
cf restage $YOUR_APP_NAME

By default, the Agent collects logs from stdout/stderr and listens to TCP port 10514. It is possible to ask the Agent to listen on a different TCP port if you are streaming logs from your application in TCP. To disable log collection from stdout/stderr, use the following configuration:

# override the TCP port
cf set-env $YOUR_APP_NAME DD_LOGS_CONFIG_TCP_FORWARD_PORT 10514
# disable log collection on stdout/stderr
cf set-env $YOUR_APP_NAME DISABLE_STD_LOG_COLLECTION true

Build

To build this buildpack, edit the relevant files and run the ./build script. If you want to upload it, run ./upload.

DogStatsD

See the DogStatsD documentation for more information. We maintain a list of DogStatsD libraries compatible with a wide range of applications.

Monitor Your Cloud Foundry Cluster

There are two points of integration with Datadog, each of which achieves a different goal:

• Datadog Agent BOSH release - Install the Datadog Agent on every node in your deployment to track system, network, and disk metrics. Enable any other Agent checks you wish.
• Datadog Firehose Nozzle - Deploy one or more Datadog Firehose Nozzle jobs. The jobs tap into your deployment’s Loggregator Firehose and send all non-container metrics to Datadog.

These integrations are meant for Cloud Foundry deployment administrators, not end users.

Prerequisites

You must have a working Cloud Foundry deployment and access to the BOSH Director that manages it. You also need BOSH CLI to deploy each integration. You may use either major version of the CLI-v1 or v2.

Install the Datadog Agent BOSH Release

Datadog provides tarballs of the Datadog Agent packaged as a BOSH release. Upload the latest release to your BOSH Director and then install it on every node in your deployment as an addon (i.e. the same way a Director deploys the BOSH Agent to all nodes).

Upload Datadog’s release to your BOSH Director

# BOSH CLI v1
bosh upload release https://cloudfoundry.datadoghq.com/datadog-agent/datadog-agent-boshrelease-latest.tgz

# BOSH CLI v2
bosh upload-release -e <BOSH_ENV> https://cloudfoundry.datadoghq.com/datadog-agent/datadog-agent-boshrelease-latest.tgz

If you’d like to create your own release, see the Datadog Agent BOSH Release repository.

Configure the Agent as an addon in your BOSH Director

Add the following to your BOSH Director’s runtime configuration file (e.g. runtime.yml):

---
releases:
  - name: datadog-agent
    version: <VERSION_YOU_UPLOADED> # specify the real version, i.e. x.y.z, not 'latest'

addons:
- name: datadog
  jobs:
  - name: dd-agent
    release: datadog-agent
  properties:
    dd:
      use_dogstatsd: true
      dogstatsd_port: 18125               # Many CF deployments have a StatsD already on port 8125
      api_key: <DD_API_KEY>
      tags: ["cloudfoundry_deployment_1"] # any tags you wish
      generate_processes: true            # to enable the process check

To see which datadog-agent release version you uploaded earlier, run bosh releases.

Load the runtime.yml

Check if you have a previously configured runtime-config by running:

# BOSH CLI v1
`bosh runtime-config`

# BOSH CLI v2
bosh -e <BOSH_ENV> runtime-config

In Bosh v2, if the runtime.yml file is empty, you should see the response: No runtime config.

Enable extra Agent checks

For each extra Agent check you want to enable across your deployment, add its configuration under the properties.dd.integrations key, e.g.:

  properties:
    dd:
      integrations:
        directory:
          init_config: {}
          instances:
            directory: "."
        #process:
        #  init_config: {}
        #...

The configuration under each check name should look the same as if you were configuring the check in its own file in the Agent’s conf.d directory.

You shouldn’t configure a check for a subset of nodes in your deployment; everything you configure in runtime.yml applies to every node.

To customize configuration for the default checks-system, network, disk, and ntp-see the full list of configuration options for the Datadog Agent BOSH release.

Sync the runtime configuration to the Director

# BOSH CLI v1
bosh update runtime-config runtime.yml

# BOSH CLI v2
bosh update-runtime-config -e <BOSH_ENV> runtime.yml

Redeploy your Cloud Foundry deployment

# BOSH CLI v1
bosh deployment yourDeploymentManifest.yml
bosh -n deploy --recreate

# BOSH CLI v2
bosh -n -d yourDeployment -e <BOSH_ENV> deploy --recreate yourDeploymentManifest.yml

Since runtime configuration applies globally, BOSH will redeploy every node in your deployment. If you have more than one deployment, redeploy those, too, to install the Datadog Agent everywhere.

Verify the Agent is installed everywhere

The easiest way to check that Agent installs were successful is to filter for them in the Host map page in Datadog. The Agent BOSH release tags each host with a generic cloudfoundry tag, so filter by that, and optionally group hosts by any tag you wish (e.g. bosh_job), as in the following screenshot:

Click on any host to zoom in, then click system within its hexagon to make sure Datadog is receiving metrics for it:

Deploy the Datadog Firehose Nozzle

As with the Datadog Agent, Datadog provides a BOSH release of the Datadog Firehose Nozzle. After uploading the release to your Director, add the Nozzle to an existing deployment, or create a new deployment that only includes the Nozzle. The instructions below assume you’re adding it to an existing Cloud Foundry deployment that has a working Loggregator Firehose.

Upload Datadog’s release to your BOSH Director

# BOSH CLI v1
bosh upload release http://cloudfoundry.datadoghq.com/datadog-firehose-nozzle/datadog-firehose-nozzle-release-latest.tgz

# BOSH CLI v2
bosh upload-release -e <BOSH_ENV> http://cloudfoundry.datadoghq.com/datadog-firehose-nozzle/datadog-firehose-nozzle-release-latest.tgz

If you’d like to create your own release, see the Datadog Firehose Nozzle release repository.

Configure a UAA client

In the manifest that contains your UAA configuration, add a new client for the Datadog Nozzle so the job(s) can access the Firehose:

uaa:
  clients:
    datadog-firehose-nozzle:
      access-token-validity: 1209600
      authorities: doppler.firehose
      authorized-grant-types: client_credentials
      override: true
      scope: doppler.firehose
      secret: <MAKE_IT_A_GOOD_ONE>

And redeploy the deployment to add the user.

Add Nozzle jobs

Configure one or more Nozzle jobs in your main Cloud Foundry deployment manifest (e.g. cf-manifest.yml):

jobs:
#- instances: 4
#  name: some_other_job
#  ...
- instances: 1            # add more instances if one job cannot keep up with the Firehose
  name: datadog_nozzle_z1
  networks:
    - name: cf1           # some network you've configured elsewhere in the manifest
  resource_pool: small_z1 # some resource_pool you've configured elsewhere in the manifest
  templates:
    - name: datadog-firehose-nozzle
      release: datadog-firehose-nozzle
  properties:
    datadog:
      api_key: <YOUR_DATADOG_API_KEY>
      api_url: https://api.datadoghq.com/api/v1/series
      flush_duration_seconds: 15 # seconds between flushes to Datadog. Default is 15.
    loggregator:
      # do NOT append '/firehose' or even a trailing slash to the URL; 'ws://<host>:<port>' will do
      traffic_controller_url: <LOGGREGATOR_URL> # e.g. ws://traffic-controller.your-cf-domain.com:8081
    nozzle:
      deployment: <DEPLOYMENT_NAME>    # tags each firehose metric with 'deployment:<DEPLOYMENT_NAME>'
      subscription_id: datadog-nozzle  # can be anything (firehose streams data evenly to all jobs using the same subscription_id)
      # disable_access_control: true   # for development only
      # insecure_ssl_skip_verify: true # for development only; enable if your UAA does not use a verifiable cert
    uaa:
      client: datadog-firehose-nozzle # client name you just configured
      client_secret: <SECRET_YOU_JUST_CONFIGURED>
      url: <UAA_URL> # e.g. https://uaa.your-cf-domain.com:8443

To see all available configuration options, check the Datadog Firehose Nozzle repository.

In the same manifest, add the Datadog Nozzle release name and version:

releases:
# - name: some-other-release
#   version: x.y.z
# ...
  - name: datadog-firehose-nozzle
    version: <VERSION_YOU_UPLOADED> # specify the real version, i.e. x.y.z, not 'latest'

To see which datadog-firehose-nozzle release version you uploaded earlier, run bosh releases.

Redeploy the deployment

# BOSH CLI v1
bosh deployment cf-manifest.yml
bosh -n deploy --recreate

# BOSH CLI v2
bosh -n -d cf-manifest -e <BOSH_ENV> deploy --recreate cf-manifest.yml

Verify the Nozzle is collecting

On the Metrics explorer page in Datadog, search for metrics beginning cloudfoundry.nozzle:

Data Collected

Metrics

The following metrics are sent by the Datadog Firehose Nozzle (cloudfoundry.nozzle). The Datadog Agent release does not send any special metrics of its own, just the usual metrics from any Agent checks you configure in the Director runtime config (and, by default, system, network, disk, and ntp metrics).

The Datadog Firehose Nozzle only collects CounterEvents (as metrics, not events) and ValueMetrics; it ignores LogMessages, Errors, and ContainerMetrics.