Connect to Google Cloud Platform to see all your Google Compute Engine (GCE) hosts in Datadog. You can see your hosts in the infrastructure overview in Datadog and sort through them, since Datadog automatically tags them with GCE host tags and any GCE labels you may have added.
Datadog's GCP integration is built to collect ALL metrics from Google Cloud Logging. Datadog strives to continually update the docs to show every sub-integration, but cloud services rapidly release new metrics and services so the list of integrations are sometimes lagging.
The Datadog <> Google Cloud integration uses Service Accounts to create an API connection between Google Cloud and Datadog. Below are instructions for creating a service account and providing Datadog with service account credentials to begin making API calls on your behalf.
Click Create credentials (near the top) and select Service account.
Give the service account a unique name and click Create.
Add the following roles: Compute Viewer, Monitoring Viewer, and Cloud Asset Viewer. Click Done.
Note: You must be a Service Account Key Admin to select Compute Engine and Cloud Asset roles. All selected roles allow Datadog to collect metrics, tags, events, and user labels on your behalf.
At the bottom of the page, find your Service Accounts and select the one you just created. Click Add Key -> Create new key, and choose JSON as the type. Click Create and Save. Take note of where this file is saved, as it is needed to complete the installation.
On the Configuration tab, select Upload Key File to integrate this project with Datadog.
Optionally, you can use tags to filter out hosts from being included in this integration. Detailed instructions on this can be found below.
If you want to monitor multiple projects, use one of the following methods:
Repeat the process above to use multiple service accounts.
Use the same service account by updating the project_id in the JSON file downloaded in step 6. Then upload the file to Datadog as described in steps 7-10.
Optionally, you can limit the GCE instances that are pulled into Datadog by entering tags in the Limit Metric Collection textbox under a given project’s dropdown menu. Only hosts that match one of the defined tags are imported into Datadog. You can use wildcards (? for single character, * for multi-character) to match many hosts, or ! to exclude certain hosts. This example includes all c1* sized instances, but excludes staging hosts:
For applications running in GCE or GKE, the Datadog Agent can be used to collect logs locally. GCP service logs are collected with Google Cloud Logging and sent to a Cloud Pub/Sub with a HTTP Push forwarder. The log collection requires 5 steps:
Pub/subs are subject to Google Cloud quotas and limitations. If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics, using different filters.
To be automatically notified when you reach this quota, activate the Pub/Sub metric integration and set up a monitor on the metric gcp.pubsub.subscription.num_outstanding_messages. Filter this monitor on the subscription that exports logs to Datadog to make sure it never goes above 1000, as per the below example:
See the individual Google Cloud integration pages for metrics.
All service events generated by your Google Cloud Platform are forwarded to your Datadog event stream.
The Google Cloud Platform integration does not include any service checks.
Tags are automatically assigned based on a variety of configuration options with regards to Google Cloud Platform and the Google Compute Engine. The following tags are automatically assigned:
Additionally, Datadog collects the following as tags:
Any hosts with <key>:<value> labels.
Custom labels from Google Pub/Sub, GCE, Cloud SQL, and Cloud Storage.
Incorrect metadata for user defined gcp.logging metrics?