Set up Observability Pipelines
Observability Pipelines connects to the open-source Vector project to help you manage and monitor the flow of logs, metrics, and traces throughout your infrastructure.
To set up Observability Pipelines, first install Vector and set up Vector configurations, then connect your Vector configurations to Observability Pipelines using your Datadog API.
Run the following OS-agnostic command in the terminal, which guides you through setting up Vector:
curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | bash
For containerized production environments, run the following command to download and install Vector:
curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | bash -s -- -y
Alternatively, you can also install Vector based on your package manager or for your specific platform and operating system.
Run the command
vector --version to make sure the installation was successful.
Using package managers
Vector supports a wide variety of package managers across several operating systems and platforms. Install Vector using your preferred package manager:
Specific platforms and operating systems
Many users choose to install Vector on their Kubernetes platform, but Vector can be installed on any of the following platforms and operating systems:
For more detailed, platform specific instructions, see Vector’s documentation.
Set up Vector configurations
Vector topologies are defined by a configuration file that tells it how to collect, transform, and route data. Vector topologies are made up of three types of components: sources, transforms, and sinks.
Vector configurations support TOML, YAML, and JSON. The location of your Vector configuration file depends on your installation method, but for most Linux-based systems, the file can be found at
If you already have Vector configurations set up, skip to Connect Vector to Observability Pipelines.
To set up a simple pipeline as an example, create a configuration file called
vector.yaml with the following:
A Vector configuration file can contain multiple Vector components. Each component is prefixed with the component type and has a unique ID. In this example, the first component has
sources for a source component with the ID
in. The component,
sources.in, uses the
stdin source, which tells Vector to receive data over
The second component,
sinks.out, uses a console sink, which tells Vector to print the data to stdout. The
encoding.codec option tells Vector to print data as plain text (unencoded).
inputs option of the
sinks.out component tells Vector where this sink’s events are coming from. In this case, events are received from the other component,
sources with ID
Run the command
vector --config ./vector.yaml to start Vector with the configuration file.
Run the following command to pipe a single event,
Hello world!, through the configuration:
echo 'Hello world!' | vector --config ./vector.yaml
See Vector Configurations for more examples on setting up the three main Vector configuration components: sources, transforms, and sinks.
Connect Vector to Observability Pipelines
Connect your Vector configuration to Observability Pipelines by doing the following:
- In Datadog, create a service account, if you don’t already have one. A service account is required to generate the application keys necessary to connect Vector to Observability Pipelines.
- Go to Observability Pipelines.
- Click Create Configuration, and follow the in-app instructions to set up the configuration.
Once the setup is complete, Vector is connected to Observability Pipelines and the data shows up in the Datadog app.
Additional helpful documentation, links, and articles: