Create a new integration

To consider an Agent-based integration complete, and thus ready to be included in the core repository and bundled with the Agent package, a number of prerequisites must be met:

• A README.md file with the right format
• A battery of tests verifying metrics collection
• A metadata.csv file listing all of the collected metrics
• A complete manifest.json file
• If the integration collects Service Checks, the service_checks.json must be complete as well

These requirements are used during the code review process as a checklist. This documentation covers the requirements and implementation details for a brand new integration.

Prerequisites

• Python 3.7+ needs to be available on your system; Python 2.7 is optional but recommended.
• Docker to run the full test suite.

In general, creating and activating Python virtual environments to isolate the development environment is good practice; however, it is not mandatory. For more information, see the Python Environment documentation.

Setup

Clone the integrations-extras repository. By default, that tooling expects you to be working in the $HOME/dd/ directory—this is optional and can be adjusted via configuration later.

mkdir $HOME/dd && cd $HOME/dd       # optional
git clone https://github.com/DataDog/integrations-extras.git

Developer toolkit

The Developer Toolkit is comprehensive and includes a lot of functionality. Here’s what you need to get started:

pip install "datadog-checks-dev[cli]"

If you chose to clone this repository to somewhere other than $HOME/dd/, you’ll need to adjust the configuration file:

ddev config set extras "/path/to/integrations-extras"

If you intend to work primarily on integrations-extras, set it as the default working repository:

ddev config set repo extras

Note: If you do not do this step, you’ll need to use -e for every invocation to ensure the context is integrations-extras:

ddev -e COMMAND [OPTIONS]

Scaffolding

One of the developer toolkit features is the create command, which creates the basic file and path structure (or “scaffolding”) necessary for a new integration.

Dry-run

Let’s try a dry-run using the -n/--dry-run flag, which won’t write anything to disk.

ddev create -n awesome

This will display the path where the files would have been written, as well as the structure itself. For now, just make sure that the path in the first line of output matches your Extras repository.

Interactive mode

The interactive mode is a wizard for creating new integrations. By answering a handful of questions, the scaffolding will be set up and lightly pre-configured for you.

ddev create awesome

After answering the questions, the output will match that of the dry-run above, except in this case the scaffolding for your new integration will actually exist!

Write the check

Intro

A Check is a Python class with the following requirements:

• If running with Agent v7+ it should be Python 3 compatible, Python 2 otherwise.
• It must derive from AgentCheck
• It must provide a method with this signature: check(self, instance)

Checks are organized in regular Python packages under the datadog_checks namespace, so your code should live under awesome/datadog_checks/awesome. The only requirement is that the name of the package has to be the same as the check name. There are no particular restrictions on the name of the Python modules within that package, nor on the name of the class implementing the check.

Implement check logic

Let’s say we want to create a Service Check named awesome.search that searches for a string on a web page. It will result in OK if the string is present, WARNING if the page is accessible but the string was not found, and CRITICAL if the page is inaccessible.

The code contained within awesome/datadog_checks/awesome/awesome.py would look something like this:

import requests

from datadog_checks.base import AgentCheck, ConfigurationError


class AwesomeCheck(AgentCheck):
    """AwesomeCheck derives from AgentCheck, and provides the required check method."""

    def check(self, instance):
        url = instance.get('url')
        search_string = instance.get('search_string')

        # It's a very good idea to do some basic sanity checking.
        # Try to be as specific as possible with the exceptions.
        if not url or not search_string:
            raise ConfigurationError('Configuration error, please fix awesome.yaml')

        try:
            response = requests.get(url)
            response.raise_for_status()
        # Something went horribly wrong
        except Exception as e:
            # Ideally we'd use a more specific message...
            self.service_check('awesome.search', self.CRITICAL, message=str(e))
        # Page is accessible
        else:
            # search_string is present
            if search_string in response.text:
                self.service_check('awesome.search', self.OK)
            # search_string was not found
            else:
                self.service_check('awesome.search', self.WARNING)

To learn more about the base Python class, see the Python API documentation.

Writing tests

There are two basic types of tests: unit tests for specific functionality, and integration tests that execute the check method and verify proper metrics collection. Tests are required if you want your integration to be included in integrations-extras. Note that pytest and tox are used to run the tests.

For more information, see the Datadog Checks Dev documentation.

Unit test

The first part of the check method retrieves and verifies two pieces of information we need from the configuration file. This is a good candidate for a unit test. Open the file at awesome/tests/test_awesome.py and replace the contents with something like this:

import pytest

# Don't forget to import your integration!
from datadog_checks.awesome import AwesomeCheck
from datadog_checks.base import ConfigurationError


@pytest.mark.unit
def test_config():
    instance = {}
    c = AwesomeCheck('awesome', {}, [instance])

    # empty instance
    with pytest.raises(ConfigurationError):
        c.check(instance)

    # only the url
    with pytest.raises(ConfigurationError):
        c.check({'url': 'http://foobar'})

    # only the search string
    with pytest.raises(ConfigurationError):
        c.check({'search_string': 'foo'})

    # this should not fail
    c.check({'url': 'http://foobar', 'search_string': 'foo'})

pytest has the concept of markers that can be used to group tests into categories. Notice we’ve marked test_config as a unit test.

The scaffolding has already been set up to run all tests located in awesome/tests. Run the tests:

ddev test awesome

Building an integration test

This test doesn’t check our collection logic though, so let’s add an integration test. We use docker to spin up an Nginx container and let the check retrieve the welcome page. Create a compose file at awesome/tests/docker-compose.yml with the following contents:

version: '3'

services:
  nginx:
    image: nginx:stable-alpine
    ports:
      - "8000:80"

Now, open the file at awesome/tests/conftest.py and replace the contents with something like this:

import os

import pytest

from datadog_checks.dev import docker_run, get_docker_hostname, get_here

URL = 'http://{}:8000'.format(get_docker_hostname())
SEARCH_STRING = 'Thank you for using nginx.'
INSTANCE = {'url': URL, 'search_string': SEARCH_STRING}


@pytest.fixture(scope='session')
def dd_environment():
    compose_file = os.path.join(get_here(), 'docker-compose.yml')

    # This does 3 things:
    #
    # 1. Spins up the services defined in the compose file
    # 2. Waits for the url to be available before running the tests
    # 3. Tears down the services when the tests are finished
    with docker_run(compose_file, endpoints=[URL]):
        yield INSTANCE


@pytest.fixture
def instance():
    return INSTANCE.copy()

Integration test

Finally, add an integration test to our awesome/tests/test_awesome.py file:

@pytest.mark.integration
@pytest.mark.usefixtures('dd_environment')
def test_service_check(aggregator, instance):
    c = AwesomeCheck('awesome', {}, [instance])

    # the check should send OK
    c.check(instance)
    aggregator.assert_service_check('awesome.search', AwesomeCheck.OK)

    # the check should send WARNING
    instance['search_string'] = 'Apache'
    c.check(instance)
    aggregator.assert_service_check('awesome.search', AwesomeCheck.WARNING)

Run only integration tests for faster development using the -m/--marker option:

ddev test -m integration awesome

The check is almost done. Let’s add the final touches by adding the integration configurations.

Configuration

Populate the README

The awesome/README.md file provided by our scaffolding already has the correct format. You must fill out the document with the relevant information.

Configuration file

When preparing a new integration, you must include an example configuration that contains the necessary options and reasonable defaults. The example configuration file, which in this case is located at awesome/datadog_checks/awesome/data/conf.yaml.example, has two top-level elements: init_config and instances. The configuration under init_config is applied to the integration globally, and is used in every instantiation of the integration, whereas anything within instances is specific to a given instantiation.

Configuration blocks in either section take the following form:

## @<COMMAND> [- <ARGS>]
## <DESCRIPTION LINE 1>
## <DESCRIPTION LINE 2>
#
<KEY>: <VALUE>

Configuration blocks follow a few guidelines:

• Description must not be empty
• Placeholders should always follow this format: <THIS_IS_A_PLACEHOLDER>, as per the documentation contributing guidelines:
• All required parameters are not commented by default.
• All optional parameters are commented by default.
• If a placeholder has a default value for an integration (like the status endpoint of an integration), it can be used instead of a generic placeholder.

@param specification

Practically speaking, the only command is @param, which is used to describe configuration blocks—primarily for documentation purposes. @param is implemented using one of the following forms:

@param <name> - <type> - required
@param <name> - <type> - optional
@param <name> - <type> - optional - default: <defval>

Arguments:

• name: the name of the parameter, e.g. search_string (mandatory).
• type: the data type for the parameter value (mandatory). Possible values:
  • boolean
  • string
  • integer
  • double
  • float
  • dictionary
  • list*
  • object
• defval: default value for the parameter; can be empty (optional).

list and object variables span over multiple lines and have special rules.

• In a list, individual elements are not documented with the @param specification
• In an object you can choose to either document elements individually with the @param specification or to have a common top-level description following the specification of the object itself.

Optional parameters

An optional parameter must be commented by default. Before every line the parameter spans on, add # (note the space) with the same indentation as the @param specification.

Block comments

You can add a block comment anywhere in the configuration file with the following rules:

• Comments start with ## (note the space)
• Comments should be indented like any variable (the hyphen doesn’t count)

Example configuration

The awesome/datadog_checks/awesome/data/conf.yaml.example for the Awesome service check:

init_config:
  ## Block comment in the init_config
  ## part.

## Block comment outside
## of the init_config part.

instances:

    ## @param url - string - required
    ## The URL you want to check
    ## (Note the indentation with the hyphen)
    #
  - url: http://example.org

    ## @param search_string - string - required
    ## The string to search for
    #
    search_string: "Example Domain"

    ## @param user - object - optional
    ## The user should map to the structure
    ## {'name': ['<FIRST_NAME>', '<LAST_NAME>'], 'username': <USERNAME>, 'password': <PASSWORD>}
    #
    # user:
    #   name:
    #     - <FIRST_NAME>
    #     - <LAST_NAME>
    #   username: <USERNAME>
    #   password: <PASSWORD>

    ## @param options - object - required
    ## Optional flags you can set
    #
    options:

      ## @param follow_redirects - boolean - optional - default: false
      ## Set to true to follow 301 Redirect
      #
      # follow_redirects: false

For more information about YAML syntax, see Wikipedia. Feel free to play around with the Online YAML Parser, too!

Manifest file

Every integration contains a manifest.json file that describes operating parameters, positioning within the greater Datadog integration eco-system, and other such items.

The complete list of mandatory and optional attributes for the manifest.json file:

Example manifest config

Our example integration has a very simple awesome/manifest.json, the bulk of which is generated by the tooling. Note that the guid must be unique (and valid), so do not use the one from this example—the tooling will generate one for you in any case.

{
  "display_name": "awesome",
  "maintainer": "email@example.org",
  "manifest_version": "1.0.0",
  "name": "awesome",
  "metric_prefix": "awesome.",
  "metric_to_check": "",
  "creates_events": false,
  "short_description": "",
  "guid": "x16b8750-df1e-46c0-839a-2056461b604x",
  "support": "contrib",
  "supported_os": [
    "linux",
    "mac_os",
    "windows"
  ],
  "public_title": "Datadog-awesome Integration",
  "categories": [
    "web"
  ],
  "type": "check",
  "is_public": false,
  "integration_id": "awesome",
  "assets": {
    "dashboards": {
      "Awesome Overview": "assets/dashboards/overview.json",
      "Awesome Investigation Dashboard": "assets/dashboards/investigation.json"
    },
    "monitors": {},
    "service_checks": "assets/service_checks.json"
  }
}

Metrics metadata file

The metadata.csv file describes all of the metrics that can be collected by the integration.

Descriptions of each column of the metadata.csv file:

Example metadata config

Our example integration doesn’t send any metrics, so in this case the generated awesome/metadata.csv contains only the row containing CSV column names.

Service check file

The service_check.json file describes the service checks made by the integration.

The service_checks.json file contains the following mandatory attributes:

Example service check config

Our example integration contains a service check, so we need to add it to the awesome/assets/service_checks.json file:

[
    {
        "agent_version": "6.0.0",
        "integration": "awesome",
        "check": "awesome.search",
        "statuses": ["ok", "warning", "critical"],
        "groups": [],
        "name": "Awesome search!",
        "description": "Returns `CRITICAL` if the check can't access the page, `WARNING` if the search string was not found, or `OK` otherwise."
    }
]

Building

setup.py provides the setuptools setup script that helps us package and build the wheel. To learn more about Python packaging, take a look at the official Python documentation.

Once your setup.py is ready, create a wheel:

• With the ddev tooling (recommended): ddev release build <INTEGRATION_NAME>
• Without the ddev tooling: cd <INTEGRATION_DIR> && python setup.py bdist_wheel

What’s in the wheel?

The wheel contains only the files necessary for the functioning of the integration itself. This includes the Check itself, the configuration example file, and some artifacts generated during the build of the wheel. All of the other elements, including the metadata files are not meant to be contained within the wheel. These latter elements are used elsewhere by the greater Datadog platform and eco-system.

Installing

The wheel is installed via the Agent integration command, available in Agent v6.10.0 and up. Depending on your environment, you may need to execute this command as a specific user or with particular privileges:

Linux (as dd-agent):

sudo -u dd-agent datadog-agent integration install -w /path/to/wheel.whl

OSX (as admin):

sudo datadog-agent integration install -w /path/to/wheel.whl

Windows (Ensure that your shell session has administrator privileges):

For Agent versions <= 6.11:

"C:\Program Files\Datadog\Datadog Agent\embedded\agent.exe" integration install -w /path/to/wheel.whl

For Agent versions >= 6.12:

"C:\Program Files\Datadog\Datadog Agent\bin\agent.exe" integration install -w /path/to/wheel.whl