Agent v6 Changes

The Agent’s main configuration file has transitioned from INI to YAML format to support complex configurations and provide a consistent experience across the Agent and checks.

Agent v5 datadog.conf –> Agent v6 datadog.yaml

To transition between Agent configuration paths and formats, use the Agent command:

sudo -u dd-agent -- datadog-agent import

This command parses an existing datadog.conf and converts supported parameters to the new format in datadog.yaml. The command also copies configuration files for checks that are currently enabled. For more details, see Upgrade to Datadog Agent v6.

Options

The following Agent configuration options were changed or removed in Agent v6. Configuration options removed were either superseded by other options, or related to features that work differently from previous versions.

Changed

Removed

Agent v6 loads any valid YAML file in: <AGENT_DIRECTORY>/conf.d/<CHECK_NAME>.d/. This enables complex configurations to be broken down into multiple files.

For example, configuration files for the http_check could be:

/etc/datadog-agent/conf.d/http_check.d/
├── backend.yaml
└── frontend.yaml

The Agent doesn’t load configuration files from any sub-directories within the <CHECK_NAME>.d folder. For example, this configuration is NOT loaded:

/etc/datadog-agent/conf.d/http_check.d/prod.d/
├── backend.yaml

Autodiscovery template files (auto_conf.yaml) are stored in the configuration folder as well. This is an example of the redisdb check configuration folder:

/etc/datadog-agent/conf.d/redisdb.d/
├── auto_conf.yaml
└── conf.yaml.example

The YAML files within a <CHECK_NAME>.d folder can have any name, as long as they have a .yaml or .yml extension. The standard name is conf.yaml.

To keep backwards compatibility, the Agent picks up configuration files in the form: <AGENT_DIRECTORY>/conf.d/<CHECK_NAME>.yaml but migrating to the updated layout is strongly recommended.

Configuration options

Agent v6 supports the following options in a check’s instance section:

Most of the environment variables used in Agent v6 are different from previous versions. See the list of environment variables for Agent v6.

Note: DD_TAGS is the same tag but in Agent v6, the format is space-separated. Previous versions were comma-separated, example for v6: DD_TAGS="simple-tag-0 tag-key-1:tag-value-1"

Proxies

For v6.4.0+, the Agent proxy settings can be overridden with the following environment variables:

The standard environment variables (HTTP_PROXY, HTTPS_PROXY, and NO_PROXY) are supported in Agent v6, but using the DD_PROXY_* variables is recommended. DD_PROXY_* variables have precedence over the other proxy variables.

The precedence order of Agent v6 proxy options is different from previous versions:

• Agent v6 uses the environment variables first and then the configuration file.
• Agent v6 overrides the values from the configuration file with the ones in the environment. For example, if both proxy.http and proxy.https are set in the configuration file but only DD_PROXY_HTTPS is set in the environment, the Agent uses the https value from the environment and the http value from the configuration file.

There are differences in hostname resolution between Agent v5 and Agent v6. For details, see the dedicated documentation.

The major changes for Agent v6 on Linux are:

• Only the Agent’s lifecycle commands (start/stop/restart/status) should be run with sudo service/sudo initctl/sudo systemctl.
• All other commands must be invoked on the Agent binary, located in the PATH (/usr/bin) as datadog-agent by default. The dd-agent command is no longer available.
• The info sub-command has been renamed to status.
• Agent v6 does not ship a SysV-init script (previously located at /etc/init.d/datadog-agent).

Service lifecycle commands

The lifecycle commands didn’t change if the service wrapper command is available on your system. For example, on Ubuntu, the lifecycle commands are:

If the service wrapper command is not available on your system, use:

• On upstart-based systems: sudo start/stop/restart/status datadog-agent
• On systemd-based systems: sudo systemctl start/stop/restart/status datadog-agent

If you’re unsure which init system your distribution uses by default, refer to the table below:

Agent commands

With Agent v6+, other functionalities are provided by the Agent binary itself as sub-commands and shouldn’t be invoked with service/systemctl/initctl. Here are a few examples:

The major changes for Agent v6 on Windows are:

• The Agent v5 Windows Agent Manager GUI was replaced with a browser-based, cross-platform manager. For details, see the Datadog Agent Manager for Windows.
• The main executable file is agent.exe (previously ddagent.exe).
• Commands should be run with the command line "%PROGRAMFILES%\datadog\datadog agent\embedded\agent.exe" <COMMAND> from an Administrator command prompt.
• The Windows service is now started as “Automatic-Delayed”. It is started automatically on boot, but after all other services. This results in a small delay in reporting metrics after a reboot.
• The Windows GUI and Windows system tray icon are now implemented separately. See the Datadog Agent Manager for Windows for more details.

The major changes for Agent v6 on MacOS are:

• The lifecycle commands (formerly datadog-agent start/stop/restart/status) are replaced by launchctl commands on the com.datadoghq.agent service, and should be run under the logged-in user. For these commands, you can also use the Datadog Agent systray app.
• All the other commands can still be run with the datadog-agent command located in the PATH (/usr/local/bin/) by default.
• The info command has been renamed to status.
• The configuration GUI is now a web-based application, which can be accessed by running the command datadog-agent launch-gui or using the systray app.

Example changes:

The APM Agent is shipped by default with Agent v6 for Linux, MacOS, and Windows packages.

The APM Agent is enabled by default on Linux. To enable it on other platforms or disable it on Linux, update the apm_config key in your datadog.yaml:

apm_config:
  enabled: true

For the Docker image, the APM Agent is disabled by default. Enable it by setting DD_APM_ENABLED to true. It listens to all interfaces by default. If you want to listen to non-local traffic on any other platform, set DD_APM_NON_LOCAL_TRAFFIC to true. For more details, see Tracing Docker Applications.

The Process Agent is shipped by default with Agent v6 for the Linux packages only.

The Process Agent is not enabled by default. To enable it, update your datadog.yaml file with the following:

process_config:
  enabled: "true"

The enabled value is a string with the following options:

• "true": Enable the Process Agent to collect processes and containers.
• "false": Only collect containers if available (the default).
• "disabled": Don’t run the Process Agent.

For Agent v6, Docker versions 1.12.1+ are supported.

The Docker check was rewritten in Go to take advantage of the internal architecture of the Agent. Therefore, the Python version (docker_daemon) is deprecated.

The new check is named docker. The Agent import command imports settings from the legacy docker_daemon.yaml configuration. All features are ported, except the following:

• url, api_version, and tags* are deprecated. It is recommended to use the standard docker environment variables.
• ecs_tags, performance_tags, and container_tags are deprecated. All relevant tags are collected by default.
• Using collect_container_count to enable the docker.container.count metric is not supported. Use docker.containers.running and .stopped.

Some options have moved from docker_daemon.yaml to the main datadog.yaml:

• collect_labels_as_tags was renamed to docker_labels_as_tags and supports high cardinality tags. See Tag Assignment and Extraction for details.
• exclude and include were renamed to ac_include and ac_exclude. To make filtering consistent across all components of the Agent, there is no longer filtering on arbitrary tags. Tag filtering is only supported for image (image name) and name (container name). Regexp filtering is still available. See Container Discovery Management for details.
• The docker_root option has been split in two options: container_cgroup_root and container_proc_root.
• exclude_pause_container was added to exclude pause containers on Kubernetes and Openshift (defaults to true).

For Agent v6, Kubernetes versions 1.3+ are supported.

The Kubernetes integration provides insights by combining:

• The kubelet check metrics from the kubelet.
• The kubernetes_apiserver check events and service checks from the API server.

The Agent import command (v6.2+) imports settings from the legacy kubernetes.yaml configuration. The following options are deprecated:

• API Server credentials (api_server_url, apiserver_client_crt, apiserver_client_key, apiserver_ca_cert): Instead, provide a kubeconfig file to the Agent with kubernetes_kubeconfig_path.
• use_histogram: Contact Datadog support to determine the best alternative for you.
• namespaces and namespace_name_regexp: Agent v6 collects metrics from all available namespaces.

The upgraded logic enables Prometheus metric collection compatible with Kubernetes versions 1.7.6+. If you run an older version or want to revert to the cadvisor collection logic, set cadvisor_port to 4194 (the port where your kubelet exposes cadvisor).

The kubernetes_state check works with Agent v5 or Agent v6.

Tagging

While Agent v5 automatically collected every pod label as tags, Agent v6 needs a whitelist. This is done with the kubernetes_pod_labels_as_tags option in datadog.yaml. See Tag Assignment and Extraction for details.

The following options and tags are deprecated:

• label_to_tag_prefix is superseded by kubernetes_pod_labels_as_tags.
• container_alias tags are not collected.
• kube_replicate_controller is only added if the pod is created by a replication controller. Instead, use the relevant creator tag (kube_deployment, kube_daemon_set, etc.).

Agent v6 ships JMXFetch, with the following changes:

JMXTerm JAR

Agent v6 does not ship the jmxterm JAR. To download and use jmxterm, refer to the upstream project.

Troubleshooting commands

Troubleshooting command syntax has changed. These commands are available for v6.2.0+, for earlier versions, refer to the JMX Agent troubleshooting:

Note: By default, these commands run on all the configured JMX checks. To specify a check, use the --checks flag, for example: sudo datadog-agent jmx list collected --checks tomcat

Only affects Windows Agents

For the Windows Agent v5, system.mem.pagefile.* metrics display inconsistent units (off by 10^6).

This problem is fixed on the Windows Agent v6. However, the Agent v5 discrepancy remains for backwards compatibility. Therefore, reported values (and associated monitors) are different when upgrading from Agent v5 to Agent v6.

When using Kubernetes, Autodiscovery sources information from the kubelet, instead of Docker daemon. This allows Autodiscovery to work without access to the Docker socket. Also, the default behavior is to source Autodiscovery templates from pod annotations. You can enable the docker config-provider to use container labels, and replace the kubelet listener by the kubelet one if you need Autodiscovery on containers running out of pods.

When specifying Autodiscovery templates in pod annotations, the annotation name prefix is ad.datadoghq.com/. The previous annotation prefix (service-discovery.datadoghq.com/) is still supported for Agent v6 but support will be removed in future versions.

Autodiscovery templates in Docker labels work with the same name prefix com.datadoghq.ad.*.

The identifier override label has been renamed from com.datadoghq.sd.check.id to com.datadoghq.ad.check.id for consistency. The previous name is still supported for Agent v6 but support will be removed in future versions.

Even though Agent v6 fully supports Python checks, some of the official Agent v5 integrations were removed or superseded:

• agent_metrics - removed
• docker_daemon - superseded by the docker check
• kubernetes - superseded by kubernetes checks

The base class for Python checks (AgentCheck) is now imported from datadog_checks.base.checks. There are a number of things that have been removed or changed in the class API. In addition, each check instance is now its own instance of the class. So you cannot share state between them.

The following methods in the AgentCheck class are not implemented:

• service_metadata
• get_service_metadata
• generate_historate_func
• generate_histogram_func
• stop

The function signature of metric senders changed:

# Previous versions
gauge(self, metric, value, tags=None, hostname=None, device_name=None, timestamp=None)

# Agent v6
gauge(self, name, value, tags=None, hostname=None, device_name=None)

The following methods have been permanently removed from AgentCheck:

• _roll_up_instance_metadata
• instance_count
• is_check_enabled
• read_config
• set_check_version
• set_manifest_path
• _get_statistic_name_from_method
• _collect_internal_stats
• _get_internal_profiling_stats
• _set_internal_profiling_stats
• get_library_versions
• get_library_info
• from_yaml
• get_service_checks
• has_warnings
• get_metrics
• has_events
• get_events

Note: All official integrations were updated to remove obsolete methods, so these changes only affect custom checks.

Precedence

For Agent v6, the official checks take precedence over custom checks (checks in <AGENT_DIRECTORY>/checks.d). Custom checks with the same name as official checks are ignored.

To fix your custom check setup with Agent v6, rename affected custom checks to a new, unused name, and rename the related .yaml configuration files accordingly.

Dependencies

If you use custom checks, there’s a chance your code depends on Python code no longer bundled with Agent v6. These packages are no longer bundled with the Agent:

• backports.ssl-match-hostname
• datadog
• decorator
• future
• futures
• google-apputils
• pycurl
• pyOpenSSL
• python-consul
• python-dateutil
• python-etcd
• python-gflags
• pytz
• PyYAML
• rancher-metadata
• tornado
• uptime
• websocket-client

If your code depends on any of these packages, install missing packages by running:

sudo -u dd-agent -- /opt/datadog-agent/embedded/bin/pip install <PACKAGE_NAME>

Similarly, you may have added a PIP package to meet a requirement for a custom check while on Agent v5. If the added PIP package had inner dependencies with packages already bundled with Agent v5 (see list above), those dependencies are missing after upgrading to Agent v6. Install the missing dependencies as described above.