Datadog-Google App Engine Integration

Overview

Install the Google App Engine integration in your Python project to:

  • See your Google App Engine services metrics: memcache, task queues, datastores
  • See metrics about requests: display percentiles, latency, cost
  • Tag Google App Engine metrics by version and compare the performance of different versions

You can also send custom metrics to Datadog.

Setup

Installation

Ensure that Billing is enabled on your Google App Engine project to collect all metrics

  1. Change directory into to your project’s application directory.
  2. Clone our Google App Engine module

    git clone https://github.com/DataDog/gae_datadog
    
  3. Edit your project’s app.yaml file

    a. Add the Datadog handler to your app.yaml file:

    handlers:
      # Should probably be at the beginning of the list
      # so it's not clobbered by a catchall route
      - url: /datadog
        script: gae_datadog.datadog.app
    

    b. Set your API key. This should be at the top level of the file and not in the handler section.

    env_variables:
      DATADOG_API_KEY: 'YOUR_API_KEY_HERE'
    

    c. Since the dogapi module sends metrics and events through a secure TLS connection, add the ssl module in the app.yaml:

    libraries:
      - name: ssl
        version: "latest"
    
  4. Add dogapi to the requirements.txt file.

    echo dogapi >> requirements.txt
    
  5. Ensure the requirements are installed.

    pip install -r requirements.txt -t lib/
    
  6. Deploy your application. Refer to the Google App Engine documentation for language specific deployment command. For Python apps, it’s:

    appcfg.py -A <project id> update app.yaml
    
  7. Enter the URL for your application in the first text box on the integration configuration screen. If you are using Task queues in the Google Developers Console, you can add them here as well.

At this point you will get a number of metrics for your environment. You can also choose to further instrument your app using the library for whatever language your app is written in.

See the Libraries page for a list of all official and community-contributed API and DogStatsD client libraries.

Data Collected

Metrics

gcp.gae.memcache.byte_hits
(gauge)
Sum of key and value bytes in successful get() and contains()
shown as byte
gcp.gae.memcache.bytes
(gauge)
Sum of key and value sizes for all live entries currently in cache
shown as byte
gcp.gae.memcache.hits
(gauge)
The counter of "successful" MemcacheService.get(Object) or MemcacheService.contains(Object) operations
shown as hit
gcp.gae.memcache.items
(gauge)
Number of entries currently "alive" in the cache
shown as item
gcp.gae.memcache.misses
(gauge)
The counter of "unsuccessful" MemcacheService.get(Object) or MemcacheService.contains(Object) operations
shown as miss
gcp.gae.memcache.oldest_item_age
(gauge)
Milliseconds since last access of least-recently-used live entry
shown as millisecond
gcp.gae.requests.cost.95percentile
(gauge)
The 95th percentile estimated fractional dollar cost of this request
shown as dollar
gcp.gae.requests.cost.avg
(gauge)
The average estimated fractional dollar cost of this request
shown as dollar
gcp.gae.requests.cost.count
(count)
The number of times the cost metric was sampled
shown as request
gcp.gae.requests.cost.max
(gauge)
The maximum estimated fractional dollar cost of this request
shown as dollar
gcp.gae.requests.cost.min
(gauge)
The minimum estimated fractional dollar cost of this request
shown as dollar
gcp.gae.requests.latency.95percentile
(gauge)
The 95th percentile of time required to process request in seconds
shown as second
gcp.gae.requests.latency.avg
(gauge)
The Average time required to process request in seconds
shown as second
gcp.gae.requests.latency.count
(count)
The number of times the request latency metric was sampled
shown as request
gcp.gae.requests.latency.max
(gauge)
The maximum time required to process request in seconds
shown as second
gcp.gae.requests.latency.min
(gauge)
The minimum time required to process request in seconds
shown as second
gcp.gae.requests.mcycles.95percentile
(gauge)
The 95th percentile of machine cycles used to process request
shown as
gcp.gae.requests.mcycles.avg
(gauge)
The average number of machine cycles used to process request
shown as
gcp.gae.requests.mcycles.count
(count)
The number of times the mcycles metric was sampled
shown as
gcp.gae.requests.mcycles.max
(gauge)
The maximum number of machine cycles used to process request
shown as
gcp.gae.requests.mcycles.min
(gauge)
The minimum number of machine cycles used to process request
shown as
gcp.gae.requests.pending_time.95percentile
(gauge)
The 95th percentile of time this request spent in the pending request queue
shown as second
gcp.gae.requests.pending_time.avg
(gauge)
The Average time this request spent in the pending request queue
shown as second
gcp.gae.requests.pending_time.count
(count)
The number of times the pending request metric was sampled
shown as request
gcp.gae.requests.pending_time.max
(gauge)
The maximum time this request spent in the pending request queue
shown as second
gcp.gae.requests.pending_time.min
(gauge)
The minimum time this request spent in the pending request queue
shown as second
gcp.gae.requests.response_size.95percentile
(gauge)
The 95th percentile of size in bytes sent back to client by request
shown as byte
gcp.gae.requests.response_size.avg
(gauge)
The Average size in bytes sent back to client by request
shown as byte
gcp.gae.requests.response_size.count
(count)
The number of times the response size metric was sampled
shown as request
gcp.gae.requests.response_size.max
(gauge)
The maximum size in bytes sent back to client by request
shown as byte
gcp.gae.requests.response_size.min
(gauge)
The minimum size in bytes sent back to client by request
shown as byte
gcp.gae.task_queue.tasks
(gauge)
The number of tasks remaining in the queue
shown as task
gcp.gae.http.server.dos_intercept_count
(count)
Interceptions performed to prevent DoS attacks.
shown as occurrence
gcp.gae.http.server.quota_denial_count
(count)
Failed requests due to the app being over quota.
shown as request
gcp.gae.http.server.response_count
(count)
HTTP responses.
shown as response
gcp.gae.http.server.response_style_count
(count)
Number of HTTP responses by serve style.
shown as response
gcp.gae.memcache.centi_mcu_count
(count)
Memcache utilization as 1/100th Memcache Compute Units.
shown as unit
gcp.gae.memcache.operation_count
(count)
Count of memcache key operations.
shown as operation
gcp.gae.memcache.received_bytes_count
(count)
Number of bytes received by app from the memcache API.
shown as byte
gcp.gae.memcache.sent_bytes_count
(count)
Number of bytes sent by app through the memcache API.
shown as byte
gcp.gae.system.cpu.usage
(gauge)
CPU usage.
shown as megahertz
gcp.gae.system.instance_count
(gauge)
Number of App Engine instances.
shown as instance
gcp.gae.system.memory.usage
(gauge)
Total memory used by running instances.
shown as byte
gcp.gae.system.network.received_bytes_count
(count)
Incoming network traffic.
shown as byte
gcp.gae.system.network.sent_bytes_count
(count)
Outgoing network traffic.
shown as byte
gcp.cloudtasks.api.request_count
(count)
Number of Cloud Tasks API calls.
shown as request
gcp.cloudtasks.queue.task_attempt_count
(count)
Number of task attempts.
shown as attempt
gcp.gae.http.server.response_latencies.avg
(gauge)
Average HTTP response latency.
shown as millisecond
gcp.gae.http.server.response_latencies.samplecount
(count)
Sample Count for HTTP response latency.
shown as millisecond
gcp.gae.http.server.response_latencies.max
(gauge)
Maximum HTTP response latency.
shown as millisecond
gcp.gae.http.server.response_latencies.min
(gauge)
Minimum HTTP response latency.
shown as millisecond
gcp.gae.http.server.response_latencies.sumsqdev
(gauge)
Sum of Squared Deviation for HTTP response latency.
shown as millisecond
gcp.cloudtasks.queue.task_attempt_delays.avg
(gauge)
Average delay between each scheduled and actual attempt times.
shown as millisecond
gcp.cloudtasks.queue.task_attempt_delays.samplecount
(count)
Sample Count for delay between each scheduled and actual attempt times.
shown as millisecond
gcp.cloudtasks.queue.task_attempt_delays.max
(gauge)
Maximum delay between each scheduled and actual attempt times.
shown as millisecond
gcp.cloudtasks.queue.task_attempt_delays.min
(gauge)
Minimum delay between each scheduled and actual attempt times.
shown as millisecond
gcp.cloudtasks.queue.task_attempt_delays.sumsqdev
(gauge)
Sum of Squared Deviation for delay between each scheduled and actual attempt times.
shown as millisecond
gcp.gae.flex.cpu.reserved_cores
(gauge)
Total number of CPU cores allocated to an App Engine flexible environment version.
shown as core
gcp.gae.flex.cpu.utilization
(gauge)
The percentage of allocated CPU in use across an App Engine flexible environment version. Note that some machine types allow bursting above 100%.
shown as percent
gcp.gae.flex.disk.read_bytes_count
(count)
Count of bytes read from disk across an App Engine flexible environment version.
shown as byte
gcp.gae.flex.disk.write_bytes_count
(count)
Delta count of bytes written from disk across an App Engine flexible environment version.
shown as byte
gcp.gae.flex.network.received_bytes_count
(count)
Count of incoming network bytes across all VMs in an App Engine flexible environment version
shown as byte
gcp.gae.flex.network.sent_bytes_count
(count)
Count of outgoing network bytes across all VMs in an App Engine flexible environment version
shown as byte

This documentation verified on November 3, 2015