Aggregating Multiple Agents Using Vector

Aggregating Multiple Agents Using Vector

Overview

The Datadog Agent can be used along with Vector. In this scenario Agents send data to Vector, which aggregates data from multiple upstream Agents:

Agents -> Vector -> Datadog

This scenario differs from using a proxy since Vector is able to process data before sending it to Datadog and other destinations. Vector capabilities include:

Notes:

  • Only logs aggregation is supported.
  • Vector can directly collect logs from alternative sources. When doing so, third party logs may not include proper tagging. A convenient way to add tags, source or service values is to use the Vector Remap Language.

Configuration

Agent configuration

To send logs to Vector, update the Agent configuration file, datadog.yaml. Only the logs data type is supported. Update the following values in the datadog.yaml file:

logs_config:
  logs_dd_url: "<VECTOR_HOST>:<VECTOR_PORT>"
  logs_no_ssl: true # If TLS/SSL is not enabled on the Vector side
  use_http: true # Vector `datadog_agent` source only supports events over HTTP(S) and not raw TCP
  # use_v2_api: false # Uncomment this line if you use a version of Vector before v0.17.0

Where VECTOR_HOST is the hostname of the system running Vector and VECTOR_PORT is the TCP port on which the Vector datadog_agent source is listening.

Docker configuration

If you are using Docker, add the following to your Agent configuration file.

-e DD_LOGS_CONFIG_LOGS_DD_URL=<VECTOR_HOST>:<VECTOR_PORT>
-e DD_LOGS_CONFIG_LOGS_NO_SSL=true
-e DD_LOGS_CONFIG_USE_HTTP=true
-e DD_LOGS_CONFIG_USE_V2_API=false

Vector configuration

To receive logs from Datadog Agent, configure Vector with a datadog_agent source. To send logs to Datadog, Vector must be configured with at least one datadog_logs sink.

See the official Vector documentation for all available configuration parameters and transforms that can be applied to logs while they are processed by Vector.

Here is a configuration example that adds a tag to every log using the Vector Remap Language:

sources:
  datadog_agents:
    type: datadog_agent
    address: "[::]:8080" # The <VECTOR_PORT> mentioned above should be set to the port value used here

transforms:
  add_tags:
    type: remap
    inputs:
      - datadog_agents
    source: |
      # The `!` shorthand is used here with the `string` function, it errors if .ddtags is not a "string".
      # The .ddtags field is always expected to be a string.
      .ddtags = string!(.ddtags) + ",sender:vector"      

sinks:
  to_datadog:
    type: datadog_logs
    inputs:
       - add_tags
    default_api_key: "${DATADOG_API_KEY_ENV_VAR}"
    encoding:
      codec: json

Using Kubernetes

Using the official Datadog chart the Agent configuration settings described above can be added to the agents.customAgentConfig value. Note: agent.useConfigMap must be set to true for agents.customAgentConfig to be taken into account.

For additional details about the Datadog Helm chart, see the Kubernetes documentation.

Vector provides an official chart for aggregating data that comes with a Datadog logs source preconfigured. For more information about installing Vector using Helm, see to the official Vector documentation.

To send logs to Datadog, a datadog_logs sink need to be added to the Vector configuration. Vector’s chart can hold any valid Vector configuration in the values.yaml file using the customConfig field. To enable datadog_logs the same kind of configuration as described under Vector configuration can be directly included as-is in the Vector chart configuration.

Manipulating Datadog logs with Vector

Logs sent to Vector can benefit from the full capabilities of Vector, including Vector Remap Language for log transformations. When received by Vector, logs sent by the Datadog Agent are structured using the expected schema. When submitting logs directly to the Datadog API, see the API documentation for a complete schema description.

Logs collected by Vector from other sources can be fully enriched. VRL can be used to adjust those logs to fill relevant fields according to the expected schema.

Further Reading