Cloud Foundry

Agent Check

Overview

Any Cloud Foundry deployment can send metrics and events to Datadog. The data helps you track the health and availability of all nodes in the deployment, monitor the jobs they run, collect metrics from the Loggregator Firehose, and more. Use this page to learn how to monitor your application on Cloud Foundry and your Cloud Foundry cluster.

There are three main components for the Cloud Foundry integration with Datadog. First, use the Buildback to collect custom metrics from your applications. Then, use the BOSH Release to collect metrics from the platform. Finally, use the Loggregator Firehose Nozzle to collect all of the other metrics from your infrastructure.

For Pivotal Cloud Foundry, you have the option to install the Datadog integration tiles with Ops Manager:

• Datadog Cluster Monitoring for PCF
• Datadog Application Monitoring for PCF

Monitor Your Applications on Cloud Foundry

Use the Datadog Cloud Foundry Buildpack to monitor your Cloud Foundry application. This is a supply buildpack for Cloud Foundry that installs a Datadog DogStatsD binary and Datadog Agent in the container your app is running on.

Setup

Cloud Foundry < 1.12

Our buildpack uses the Cloud Foundry multi-buildpack feature that was introduced in version 1.12.

For older versions, Cloud Foundry provides a back-port of this feature in the form of a buildpack. You must install and configure this backport in order to use Datadog’s buildpack:

1. Upload the multi-buildpack back-port. Download the latest multi-build pack release and upload it to your Cloud Foundry environment.

   cf create-buildpack multi-buildpack ./multi-buildpack-v-x.y.z.zip 99 --enable

2. Add a multi-buildpack manifest to your application. As detailed on the multi-buildpack back-port repo, create a multi-buildpack.yml file at the root of your application and configure it for your environment. Add a link to the Datadog Cloud Foundry Buildpack and to your regular buildpack:

     buildpacks:
       - "https://cloudfoundry.datadoghq.com/datadog-cloudfoundry-buildpack/datadog-cloudfoundry-buildpack-3.1.0.zip"
       - "https://github.com/cloudfoundry/ruby-buildpack#v1.7.18" # Replace this with your regular buildpack

   The URLs for the Datadog Buildpack are:

   • https://cloudfoundry.datadoghq.com/datadog-cloudfoundry-buildpack/datadog-cloudfoundry-buildpack-latest.zip
   • https://cloudfoundry.datadoghq.com/datadog-cloudfoundry-buildpack/datadog-cloudfoundry-buildpack-x.y.z.zip

   Do not use the latest version here (replace x.y.z by the specific version you want to use).

   Important: Your regular buildpack should be the last in the manifest to act as a final buildpack. To learn more refer to cloud foundry documentation about buildpacks.

3. Push your application with the multi-buildpack Ensure that the multi-buildpack is the buildpack selected by Cloud Foundry for your application:

   cf push <YOUR_APP> -b multi-buildpack

Cloud Foundry >= 1.12

1. Upload the Datadog Cloud Foundry Buildpack. Download the latest Datadog build pack release and upload it to your Cloud Foundry environment.

   cf create-buildpack datadog-cloudfoundry-buildpack ./datadog-cloudfoundry-buildpack-latest.zip

2. Push your application with the Datadog buildpack and your buildpacks. The process to push your application with multiple buildpacks is described in the Cloud Foundry documentation.

   cf push <YOUR_APP> --no-start -b binary_buildpack
   cf v3-push <YOUR_APP> -b datadog-cloudfoundry-buildpack -b <YOUR-BUILDPACK-1> -b <YOUR-FINAL-BUILDPACK>

Important: If you were using a single buildpack before, it should be the last one loaded so it acts as a final buildpack. To learn more refer to Cloud Foundry documentation about buildpacks.

Meta-Buildpack (deprecated)

If you are a meta-buildpack user, Datadog’s buildpack can be used as a decorator out of the box.

Note: The meta-buildpack has been deprecated by pivotal in favor of the multi-buildpack and Datadog might drop support for it in a future release.

Configuration

Metric Collection

Set an API Key in your environment to enable the buildpack:

# set the environment variable
cf set-env <YOUR_APP> DD_API_KEY <DD_API_KEY>
# restage the application to make it pick up the new environment variable and use the buildpack
cf restage <YOUR_APP>

Trace Collection

The Datadog Trace Agent (APM) is enabled by default. Learn more about setup for your specific language in APM Setup.

Log Collection

Enable log collection

To start collecting logs from your application in Cloud Foundry, the Agent contained in the buildpack needs to be activated and log collection enabled.

cf set-env <YOUR_APP_NAME> RUN_AGENT true
cf set-env <YOUR_APP_NAME> DD_LOGS_ENABLED true
# Disable the Agent core checks to disable system metrics collection
cf set-env <YOUR_APP_NAME> DD_ENABLE_CHECKS false
# Redirect Container Stdout/Stderr to a local port so the Agent collects the logs
cf set-env <YOUR_APP_NAME> STD_LOG_COLLECTION_PORT <PORT>
# Configure the Agent to collect logs from the wanted port and set the value for source and service
cf set-env <YOUR_APP_NAME> LOGS_CONFIG '[{"type":"tcp","port":"<PORT>","source":"<SOURCE>","service":"<SERVICE>"}]'
# restage the application to make it pick up the new environment variable and use the buildpack
cf restage <YOUR_APP_NAME>

Configure log collection

The following parameters can be used to configure log collection:

Example:

A Java application named app01 is running in Cloud Foundry. The following configuration redirects the container stdout/stderr to the local port 10514. It then configures the Agent to collect logs from that port while setting the proper value for service and source:

# Redirect Stdout/Stderr to port 10514
cf set-env app01 STD_LOG_COLLECTION_PORT 10514
# Configure the Agent to listen to port 10514
cf set-env app01 LOGS_CONFIG '[{"type":"tcp","port":"10514","source":"java","service":"app01"}]'

Notification in case of misconfigured proxy

For Agent v6.12+, when using a proxy configuration with the Buildpack, a verification is made to check if the connection can be established. Log collection is started depending on the result of this test.

If the connection fails to be established and the log collection is not started, an event like the one below is sent to your Datadog event stream. Set up a monitor to track these events and be notified when a misconfigured Buildpack is deployed:

Build

To build this buildpack, edit the relevant files and run the ./build script. To upload it, run ./upload.

DogStatsD

See the DogStatsD documentation for more information. There is a list of DogStatsD libraries compatible with a wide range of applications.

Monitor Your Cloud Foundry Cluster

There are two points of integration with Datadog, each of which achieves a different goal:

• Datadog Agent BOSH release - Install the Datadog Agent on every node in your deployment to track system, network, and disk metrics. Enable any other Agent checks you wish.
• Datadog Firehose Nozzle - Deploy one or more Datadog Firehose Nozzle jobs. The jobs tap into your deployment’s Loggregator Firehose and send all non-container metrics to Datadog.

Prerequisites

You must have a working Cloud Foundry deployment and access to the BOSH Director that manages it. You also need BOSH CLI to deploy each integration. You may use either major version of the CLI-v1 or v2.

Install the Datadog Agent BOSH Release

Datadog provides tarballs of the Datadog Agent packaged as a BOSH release. Upload the latest release to your BOSH Director and then install it on every node in your deployment as an addon (the same way a Director deploys the BOSH Agent to all nodes).

Upload Datadog’s release to your BOSH Director

# BOSH CLI v1
bosh upload release https://cloudfoundry.datadoghq.com/datadog-agent/datadog-agent-boshrelease-latest.tgz

# BOSH CLI v2
bosh upload-release -e <BOSH_ENV> https://cloudfoundry.datadoghq.com/datadog-agent/datadog-agent-boshrelease-latest.tgz

If you’d like to create your own release, see the Datadog Agent BOSH Release repository.

Configure the Agent as an addon in your BOSH Director

Add the following to your BOSH Director’s runtime configuration file (e.g. runtime.yml):

---
releases:
  - name: datadog-agent
    version: <VERSION_YOU_UPLOADED> # specify the real version (x.y.z not 'latest')

addons:
- name: datadog
  jobs:
  - name: dd-agent
    release: datadog-agent
  properties:
    dd:
      use_dogstatsd: true
      dogstatsd_port: 18125       # Many CF deployments have a StatsD already on port 8125
      api_key: <DD_API_KEY>
      tags: ["<KEY:VALUE>"]       # any tags you wish
      generate_processes: true    # to enable the process check

To see which datadog-agent release version you uploaded earlier, run bosh releases.

Load the runtime.yml

Check if you have a previously configured runtime-config by running:

# BOSH CLI v1
`bosh runtime-config`

# BOSH CLI v2
bosh -e <BOSH_ENV> runtime-config

In Bosh v2, if the runtime.yml file is empty, you should see the response: No runtime config.

Enable extra Agent checks

For each extra Agent check you want to enable across your deployment, add its configuration under the properties.dd.integrations key, for example:

  properties:
    dd:
      integrations:
        directory:
          init_config: {}
          instances:
            directory: "."
        #process:
        #  init_config: {}
        #...

The configuration under each check name should look the same as if you were configuring the check in its own file in the Agent’s conf.d directory.

Everything you configure in runtime.yml applies to every node. You cannot configure a check for a subset of nodes in your deployment.

To customize configuration for the default checks-system, network, disk, and ntp-see the full list of configuration options for the Datadog Agent BOSH release.

Sync the runtime configuration to the Director

# BOSH CLI v1
bosh update runtime-config runtime.yml

# BOSH CLI v2
bosh update-runtime-config -e <BOSH_ENV> runtime.yml

Redeploy your Cloud Foundry deployment

# BOSH CLI v1
bosh deployment <YOUR_DEPLOYMENT_MANIFEST>.yml
bosh -n deploy --recreate

# BOSH CLI v2
bosh -n -d <YOUR_DEPLOYMENT> -e <BOSH_ENV> deploy --recreate <YOUR_DEPLOYMENT_MANIFEST>.yml

Since runtime configuration applies globally, BOSH redeploys every node in your deployment. If you have more than one deployment, redeploy all deployments to install the Datadog Agent everywhere.

Verify the Agent is installed everywhere

To check if the Agent installs were successful, filter by cloudfoundry on the Host map page in Datadog. The Agent BOSH release tags each host with a generic cloudfoundry tag. Optionally group hosts by any tag, such as bosh_job, as in the following screenshot:

Click on any host to zoom in, then click system within its hexagon to make sure Datadog is receiving metrics for it:

Deploy the Datadog Firehose Nozzle

Datadog provides a BOSH release of the Datadog Firehose Nozzle. After uploading the release to your Director, add the Nozzle to an existing deployment, or create a new deployment that only includes the Nozzle. The instructions below assume you’re adding it to an existing Cloud Foundry deployment that has a working Loggregator Firehose.

Upload Datadog’s release to your BOSH Director

# BOSH CLI v1
bosh upload release http://cloudfoundry.datadoghq.com/datadog-firehose-nozzle/datadog-firehose-nozzle-release-latest.tgz

# BOSH CLI v2
bosh upload-release -e <BOSH_ENV> http://cloudfoundry.datadoghq.com/datadog-firehose-nozzle/datadog-firehose-nozzle-release-latest.tgz

If you’d like to create your own release, see the Datadog Firehose Nozzle release repository.

Configure a UAA client

In the manifest that contains your UAA configuration, add a new client for the Datadog Nozzle so the job(s) can access the Firehose:

uaa:
  clients:
    datadog-firehose-nozzle:
      access-token-validity: 1209600
      authorities: doppler.firehose
      authorized-grant-types: client_credentials
      override: true
      scope: doppler.firehose
      secret: <YOUR_SECRET>

Redeploy the deployment to add the user.

Add Nozzle jobs

Configure one or more Nozzle jobs in your main Cloud Foundry deployment manifest (e.g. cf-manifest.yml):

jobs:
#- instances: 4
#  name: some_other_job
#  ...
- instances: 1            # add more instances if one job cannot keep up with the Firehose
  name: datadog_nozzle_z1
  networks:
    - name: cf1           # some network you've configured elsewhere in the manifest
  resource_pool: small_z1 # some resource_pool you've configured elsewhere in the manifest
  templates:
    - name: datadog-firehose-nozzle
      release: datadog-firehose-nozzle
  properties:
    datadog:
      api_key: <YOUR_DATADOG_API_KEY>
      api_url: https://api.datadoghq.com/api/v1/series
      flush_duration_seconds: 15 # seconds between flushes to Datadog. Default is 15.
    loggregator:
      # do NOT append '/firehose' or even a trailing slash to the URL; 'ws://<host>:<port>' works
      traffic_controller_url: <LOGGREGATOR_URL> # e.g. ws://traffic-controller.your-cf-domain.com:8081
    nozzle:
      deployment: <DEPLOYMENT_NAME>    # tags each firehose metric with 'deployment:<DEPLOYMENT_NAME>'
      subscription_id: datadog-nozzle  # can be anything (firehose streams data evenly to all jobs using the same subscription_id)
      # disable_access_control: true   # for development only
      # insecure_ssl_skip_verify: true # for development only; enable if your UAA does not use a verifiable cert
    uaa:
      client: datadog-firehose-nozzle # client name you just configured
      client_secret: <SECRET_YOU_JUST_CONFIGURED>
      url: <UAA_URL> # e.g. https://uaa.your-cf-domain.com:8443

To see all available configuration options, check the Datadog Firehose Nozzle repository.

In the same manifest, add the Datadog Nozzle release name and version:

releases:
# - name: <SOME_OTHER_RELEASE>
#   version: <x.y.z>
# ...
  - name: datadog-firehose-nozzle
    version: <VERSION_YOU_UPLOADED> # specify the real version (x.y.z not 'latest')

To see which datadog-firehose-nozzle release version you uploaded earlier, run bosh releases.

Redeploy the deployment

# BOSH CLI v1
bosh deployment cf-manifest.yml
bosh -n deploy --recreate

# BOSH CLI v2
bosh -n -d cf-manifest -e <BOSH_ENV> deploy --recreate cf-manifest.yml

Verify the Nozzle is collecting

On the Metrics explorer page in Datadog, search for metrics beginning cloudfoundry.nozzle:

Data Collected

Metrics

The following metrics are sent by the Datadog Firehose Nozzle (cloudfoundry.nozzle). The Datadog Agent release does not send any special metrics of its own, just the usual metrics from any Agent checks you configure in the Director runtime config (and, by default, system, network, disk, and ntp metrics).

The Datadog Firehose Nozzle only collects CounterEvents (as metrics, not events), ValueMetrics, and ContainerMetrics; it ignores LogMessages and Errors.