Aggregating Multiple Agents Using Vector
The Datadog Agent can be used along with Vector. In this scenario Agents send data
to Vector, which aggregates data from multiple upstream Agents:
Agents -> Vector -> Datadog
This scenario differs from using a proxy since Vector is able to process data before sending
it to Datadog and other destinations. Vector capabilities include:
- Only logs aggregation is supported.
- Vector can directly collect logs from alternative sources. When doing so, third party logs may not include proper tagging. A convenient way to add tags, source or service values is to use the Vector Remap Language.
To send logs to Vector, update the Agent configuration file,
Only the logs data type is supported. Update the following values in the
logs_no_ssl: true # If TLS/SSL is not enabled on the Vector side
use_http: true # Vector `datadog_logs` source only supports events over HTTP(S) and not raw TCP
# use_v2_api: false # Uncomment this line if you use a version of Vector before v0.17.0
VECTOR_HOST is the hostname of the system running Vector and
VECTOR_PORT is the TCP port on which
datadog_logs source is listening.
To receive logs from Datadog Agent, configure Vector with a datadog_logs source.
To send logs to Datadog, Vector must be configured with at least one datadog_logs sink.
See the official Vector documentation for all available configuration parameters and
transforms that can be applied to logs while they are processed by Vector.
Here is a configuration example that adds a tag to every log using the Vector Remap Language:
address: "[::]:8080" # The <VECTOR_PORT> mentioned above should be set to the port value used here
# The `!` shorthand is used here with the `string` function, it errors if .ddtags is not a "string".
# The .ddtags field is always expected to be a string.
.ddtags = string!(.ddtags) + ",sender:vector"
Using the official Datadog chart the Agent configuration settings described above can be added
agents.customAgentConfig value. Note:
agent.useConfigMap must be set to
agents.customAgentConfig to be taken into account.
For additional details about the Datadog Helm chart, see the Kubernetes documentation.
Vector provides an official chart for aggregating data that comes with a Datadog
logs source preconfigured. For more information about installing Vector using Helm,
see to the official Vector documentation.
To send logs to Datadog, a
datadog_logs sink need to be added to the Vector
configuration. Vector’s chart can hold any valid Vector configuration in the
values.yaml file using the
customConfig field. To enable
datadog_logs the same kind of configuration as
described under Vector configuration can be directly included as-is in the Vector chart configuration.
Manipulating Datadog logs with Vector
Logs sent to Vector can benefit from the full capabilities of Vector, including Vector Remap Language for log transformations.
When received by Vector, logs sent by the Datadog Agent are structured using the expected schema. When
submitting logs directly to the Datadog API, see the API documentation
for a complete schema description.
Logs collected by Vector from other sources can be fully enriched. VRL can be used to adjust those logs
to fill relevant fields according to the expected schema.
Additional helpful documentation, links, and articles: