Google Cloud Platform

Overview

Connect to Google Cloud Platform to see all your Google Compute Engine (GCE) hosts in Datadog. You can see your hosts in the infrastructure overview in Datadog and sort through them, since Datadog automatically tags them with GCE host tags and any GCE labels you may have added.

Datadog's GCP integration is built to collect all Google Cloud metrics. Datadog strives to continually update the docs to show every sub-integration, but cloud services rapidly release new metrics and services so the list of integrations are sometimes lagging.
IntegrationDescription
App EnginePaaS (platform as a service) to build scalable applications
Big QueryEnterprise data warehouse
BigtableNoSQL Big Data database service
Cloud SQLMySQL database service
Cloud APIsProgrammatic interfaces for all Google Cloud Platform services
Cloud ComposerA fully managed workflow orchestration service
Cloud DataprocA cloud service for running Apache Spark and Apache Hadoop clusters
Cloud FilestoreHigh-performance, fully managed file storage
Cloud FirestoreA flexible, scalable database for mobile, web, and server development
Cloud InterconnectHybrid connectivity
Cloud IoTSecure device connection and management
Cloud Load BalancingDistribute load-balanced compute resources
Cloud Memorystore for RedisA fully managed in-memory data store service
Cloud RouterExchange routes between your VPC and on-premises networks by using BGP
Cloud RunManaged compute platform that runs stateless containers through HTTP
Cloud TasksDistributed task queues
Cloud TPUTrain and run machine learning models
Compute EngineHigh performance virtual machines
Container EngineKubernetes, managed by google
DatastoreNoSQL database
FirebaseMobile platform for application development
FunctionsServerless platform for building event-based microservices
Kubernetes EngineCluster manager and orchestration system
Machine LearningMachine learning services
Pub/SubReal-time messaging service
SpannerHorizontally scalable, globally consistent, relational database service
Cloud LoggingReal-time log management and analysis
StorageUnified object storage
VPNManaged network functionality

Setup

Metric collection

Installation

The Datadog <> Google Cloud integration uses Service Accounts to create an API connection between Google Cloud and Datadog. Below are instructions for creating a service account and providing Datadog with service account credentials to begin making API calls on your behalf.

Note: Google Cloud billing, the Cloud Monitoring API, the Compute Engine API, and the Cloud Asset API must all be enabled for the project(s) you wish to monitor.

  1. Navigate to the Google Cloud credentials page for the Google Cloud project where you would like to setup the Datadog integration.

  2. Click Create credentials (near the top) and select Service account.

    settings

  3. Give the service account a unique name and click Create.

  4. Add the following roles: Compute Viewer, Monitoring Viewer, and Cloud Asset Viewer. Click Done.

    Note: You must be a Service Account Key Admin to select Compute Engine and Cloud Asset roles. All selected roles allow Datadog to collect metrics, tags, events, and user labels on your behalf.

  5. At the bottom of the page, find your Service Accounts and select the one you just created. Click Add Key -> Create new key, and choose JSON as the type. Click Create and Save. Take note of where this file is saved, as it is needed to complete the installation.

  6. Navigate to the Datadog Google Cloud Integration tile.

  7. On the Configuration tab, select Upload Key File to integrate this project with Datadog.

  8. Optionally, you can use tags to filter out hosts from being included in this integration. Detailed instructions on this can be found below.

    settings

  9. Press Install/Update.

  10. If you want to monitor multiple projects, use one of the following methods:

    • Repeat the process above to use multiple service accounts.
    • Use the same service account by updating the project_id in the JSON file downloaded in step 5. Then upload the file to Datadog as described in steps 6-9.

Configuration

Optionally, you can limit the GCE instances that are pulled into Datadog by entering tags in the Limit Metric Collection textbox under a given project’s dropdown menu. Only hosts that match one of the defined tags are imported into Datadog. You can use wildcards (? for single character, * for multi-character) to match many hosts, or ! to exclude certain hosts. This example includes all c1* sized instances, but excludes staging hosts:

datadog:monitored,env:production,!env:staging,instance-type:c1.*

See Google’s documentation on Creating and managing labels for more details.

Log collection

For applications running in GCE or GKE, the Datadog Agent can be used to collect logs locally. GCP service logs are collected with Google Cloud Logging and sent to a Cloud Pub/Sub with a HTTP Push forwarder. The log collection requires 5 steps:

  1. If you haven’t already, set up the Google Cloud platform integration first.
  2. Create a new Cloud Pub/Sub.
  3. Setup the Pub/Sub to forward logs to Datadog.
  4. Configure exports from Google Cloud logs to the Pub/Sub.

Warning: Pub/subs are subject to Google Cloud quotas and limitations. If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics. See the Monitor the Log Forwarding section for information on how to set up a monitor to be automatically notified if you get close to those limits.

Create a Cloud Pub Sub

  1. Go to the Cloud Pub Sub console and create a new topic.

    Create a topic

  2. Give that topic an explicit name such as export-logs-to-datadog and Save.

Forward logs to Datadog

  1. Go back to the Pub/Sub Topics overview page, and add select Subscriptions in the left hand navigation. Select Create Subscription.

  2. Create a subscription ID and select the topic you previously created.

  3. Select the Push method and enter the following: https://gcp-intake.logs./api/v2/logs?dd-api-key=<DATADOG_API_KEY>&dd-protocol=gcp.

    You can create an API key or pick an existing API key in Datadog Organization Settings -> API Keys.

  4. Configure any additional options, such as Subscription expiration, Acknowledgment deadline, Message retention duration, or Dead lettering.

  5. Hit Create at the bottom.

The Pub/Sub is ready to receive logs from Google Cloud Logging and forward them to Datadog.

Export logs from Google Cloud

  1. Go to the Logs Explorer page and filter the logs that need to be exported.

  2. From Log Router tab, select Create Sink.

  3. Provide a name for the sink.

  4. Choose Cloud Pub/Sub as the destination and select the pub/sub that was created for that purpose. Note: The pub/sub can be located in a different project.

    Export Google Cloud Pub/Sub Logs to Pub Sub

  5. Click Create Sink and wait for the confirmation message to show up.

Note: It is possible to create several exports from Google Cloud Logging to the same Pub/Sub with different sinks.

Warning: Pub/subs are subject to Google Cloud quotas and limitations. If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics. See the Monitor the Log Forwarding section for information on how to setup a monitor to be automatically notified if you get close to those limits.

Monitor the log forwarding

Pub/subs are subject to Google Cloud quotas and limitations. If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics, using different filters.

To be automatically notified when you reach this quota, activate the Pub/Sub metric integration and set up a monitor on the metric gcp.pubsub.subscription.num_outstanding_messages. Filter this monitor on the subscription that exports logs to Datadog to make sure it never goes above 1000, as per the below example:

Pub Sub monitoring

Data Collected

Metrics

See the individual Google Cloud integration pages for metrics.

Events

All service events generated by your Google Cloud Platform are forwarded to your Datadog event stream.

Service Checks

The Google Cloud Platform integration does not include any service checks.

Tags

Tags are automatically assigned based on a variety of Google Cloud Platform and Google Compute Engine configuration options. The project_id tag is added to all metrics. Additional tags are collected from the Google Cloud Platform when available, and varies based on metric type.

Additionally, Datadog collects the following as tags:

  • Any hosts with <key>:<value> labels.
  • Custom labels from Google Pub/Sub, GCE, Cloud SQL, and Cloud Storage.

Troubleshooting

Incorrect metadata for user defined gcp.logging metrics?

For non-standard gcp.logging metrics, such as metrics beyond Datadog’s out of the box logging metrics), the metadata applied may not be consistent with Google Cloud Logging.

In these cases, the metadata should be manually set by navigating to the metric summary page, searching and selecting the metric in question, and clicking the pencil icon next to the metadata.

Need help? Contact Datadog support.