Set up Observability Pipelines

Overview

Observability Pipelines connects to the open-source Vector project to help you manage and monitor the flow of logs, metrics, and traces throughout your infrastructure.

To set up Observability Pipelines, first install Vector and set up Vector configurations, then connect your Vector configurations to Observability Pipelines using your Datadog API.

Install Vector

Quick start

Terminal

Run the following OS-agnostic command in the terminal, which guides you through setting up Vector:

curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | bash

Containers

For containerized production environments, run the following command to download and install Vector:

curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | bash -s -- -y

Alternatively, you can also install Vector based on your package manager or for your specific platform and operating system.

Run the command vector --version to make sure the installation was successful.

Using package managers

Vector supports a wide variety of package managers across several operating systems and platforms. Install Vector using your preferred package manager:

Specific platforms and operating systems

Many users choose to install Vector on their Kubernetes platform, but Vector can be installed on any of the following platforms and operating systems:

For more detailed, platform specific instructions, see Vector’s documentation.

Set up Vector configurations

Vector topologies are defined by a configuration file that tells it how to collect, transform, and route data. Vector topologies are made up of three types of components: sources, transforms, and sinks.

Vector configurations support TOML, YAML, and JSON. The location of your Vector configuration file depends on your installation method, but for most Linux-based systems, the file can be found at /etc/vector/vector.toml.

If you already have Vector configurations set up, skip to Connect Vector to Observability Pipelines.

Configuration example

To set up a simple pipeline as an example, create a configuration file called vector.yaml with the following:

sources:
  in:
    type: stdin
sinks:
  out:
    inputs:
      - in
    type: console
    encoding:
      codec: text

A Vector configuration file can contain multiple Vector components. Each component is prefixed with the component type and has a unique ID. In this example, the first component has sources for a source component with the ID in. The component, sources.in, uses the stdin source, which tells Vector to receive data over stdin.

The second component, sinks.out, uses a console sink, which tells Vector to print the data to stdout. The encoding.codec option tells Vector to print data as plain text (unencoded).

The inputs option of the sinks.out component tells Vector where this sink’s events are coming from. In this case, events are received from the other component, sources with ID in.

Run the command vector --config ./vector.yaml to start Vector with the configuration file.

Run the following command to pipe a single event, Hello world!, through the configuration:

echo 'Hello world!' | vector --config ./vector.yaml

See Vector Configurations for more examples on setting up the three main Vector configuration components: sources, transforms, and sinks.

Connect Vector to Observability Pipelines

Connect your Vector configuration to Observability Pipelines by doing the following:

  • In Datadog, create a service account, if you don’t already have one. A service account is required to generate the application keys necessary to connect Vector to Observability Pipelines.
  • Go to Observability Pipelines.
  • Click Create Configuration, and follow the in-app instructions to set up the configuration.

Once the setup is complete, Vector is connected to Observability Pipelines and the data shows up in the Datadog app.

Further Reading