---
title: Advanced Worker Configurations
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: >-
  Docs > Observability Pipelines > Configuration > Install the Worker > Advanced
  Worker Configurations
---

# Advanced Worker Configurations

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). ().
{% /alert %}

{% /callout %}

## Overview{% #overview %}

This document explains bootstrapping for the Observability Pipelines Worker, other Worker configuration options, and how to enable the health check endpoint and the liveness and readiness probes.

## Bootstrap Options{% #bootstrap-options %}

{% alert level="danger" %}
All configuration file paths specified in the pipeline need to be under `/DD_OP_DATA_DIR/config`. Modifying files under that location while OPW is running might have adverse effects.
{% /alert %}

Bootstrap the Observability Pipelines Worker within your infrastructure before you set up a pipeline. These environment variables are separate from the pipeline environment variables. The location of the related directories and files:

- Default data directory: `/var/lib/observability-pipelines-worker`
- Bootstrap file: `/etc/observability-pipelines-worker/bootstrap.yaml`
- Environment variables file: `/etc/default/observability-pipelines-worker`

**Note**: `DD_OP_DATA_DIR` can only be owned by a single Observability Pipelines Worker. If you have multiple Workers, you must use unique data directories.

To set bootstrap options, do one of the following:

- Use environmental variables.
- Create a `bootstrap.yaml` and start the Worker instance with `--bootstrap-config /path/to/bootstrap.yaml`.

The following is a list of bootstrap options, their related pipeline environment variables, and whether the bootstrap value or the environment variable has a higher precedence (priority), if both have been set.

{% dl %}

{% dt %}
`api`
{% /dt %}

{% dd %}
**Pipeline environment variable**: `DD_OP_API_ENABLED`
{% /dd %}

{% dd %}
**Priority**: `DD_OP_API_ENABLED`
{% /dd %}

{% dd %}
An example configuration:
{% /dd %}

{% dd %}
`api`:`enabled`: `true``address`: `"127.0.0.1:8686" # optional`
{% /dd %}

{% dd %}
Note: Setting `address` is optional. It is the network address to which the API should bind. If you're running the Worker in a Docker container, bind to `0.0.0.0`. Otherwise, the API is not exposed outside of the container.
{% /dd %}

{% dd %}
**Description**: Enable the Observability Pipelines Worker API so you can see the Worker's processes with the `tap` or `top` command. See [Run, tap, or top the Worker](https://docs.datadoghq.com/observability_pipelines/install_the_worker/worker_commands/#run-tap-or-top-the-worker) for more information. If you are using the Helm charts provided when you [set up a pipeline](https://docs.datadoghq.com/observability_pipelines/set_up_pipelines/), then the API has already been enabled. Otherwise, make sure the environment variable `DD_OP_API_ENABLED` is set to `true` in `/etc/observability-pipelines-worker/bootstrap.yaml`. This sets up the API to listen on `localhost` and port `8686`, which is what the CLI for `tap` is expecting.See Enable liveness and readiness probe on how to expose the `/health` endpoint.
{% /dd %}

{% dt %}
`api_key`
{% /dt %}

{% dd %}
**Pipeline environment variable**: `DD_API_KEY`
{% /dd %}

{% dd %}
**Priority**: `DD_API_KEY`
{% /dd %}

{% dd %}
**Description**: Create a [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys) for this environment variable. [Remote Configuration](https://docs.datadoghq.com/remote_configuration) must be enabled for the API key. See [Security considerations](https://docs.datadoghq.com/remote_configuration/#security-considerations) for information on safeguards implemented for Remote Configuration.
{% /dd %}

{% dt %}
`data_dir`
{% /dt %}

{% dd %}
**Pipeline environment variable**: `DD_OP_DATA_DIR`
{% /dd %}

{% dd %}
**Priority**: `DD_OP_DATA_DIR`
{% /dd %}

{% dd %}
**Description**: The data directory (optional, default: `/var/lib/observability-pipelines-worker`). This is the file system directory that the Observability Pipelines Worker uses for local state.
{% /dd %}

{% dt %}
`pipeline_id`
{% /dt %}

{% dd %}
**Pipeline environment variable**: `DD_OP_PIPELINE_ID`
{% /dd %}

{% dd %}
**Priority**: `DD_OP_PIPELINE_ID`
{% /dd %}

{% dd %}
**Description**: Create an [Observability Pipelines pipeline ID](https://app.datadoghq.com/observability-pipelines) for this environment variable.
{% /dd %}

{% dt %}
`proxy`
{% /dt %}

{% dd %}
**Pipeline environment variables**: `DD_PROXY_HTTP`, `DD_PROXY_HTTPS`, `DD_PROXY_NO_PROXY`
{% /dd %}

{% dd %}
Set proxy servers for the Observability Pipelines Worker. The proxy configuration for the Worker works in the same way as it does for the [Datadog Agent](https://docs.datadoghq.com/agent/configuration/proxy/?tab=linux#environment-variables).
{% /dd %}

{% dd %}
**Priority**: The settings are applied to the entire Worker process. The HTTP proxy and HTTPS values are resolved in this order:1. `DD_PROXY_HTTP(S)`2. `HTTP(S)_PROXY`3. `proxy` :
{% /dd %}

{% dd %}
An example proxy configuration:
{% /dd %}

{% dd %}
`proxy`:`enabled`: `true``https`: `https://foo.bar:3128`
{% /dd %}

{% dd %}
**Description**: The Observability Pipelines Worker can route external requests through forward proxies, such as Squid. Forward proxies forward client requests from the Observability Pipelines Worker to the internet. You might use them as a web firewall to forbid or allow certain domains, ports, or protocols. Forward proxies usually do not terminate SSL and therefore do not have access to the request content. They only pass packets back and forth between the client and the destination. [HTTP tunnels](https://en.wikipedia.org/wiki/HTTP_tunnel) are used to secure communication through a forward proxy.
{% /dd %}

{% dd %}
**Notes**:
{% /dd %}

{% dd %}
This option is available for Observability Pipelines Worker 2.1 and later.
{% /dd %}

{% dd %}
The Observability Pipelines Worker cannot route external requests through reverse proxies, such as HAProxy and NGINX.
{% /dd %}

{% dd %}
The `DD_PROXY_HTTP(S)` and `HTTP(S)_PROXY` environment variables need to be already exported in your environment for the Worker to resolve them. They cannot be prepended to the Worker installation script.
{% /dd %}

{% dt %}
`secret`
{% /dt %}

{% dd %}
**Pipeline environment variable**: None
{% /dd %}

{% dd %}
**Priority**: N/A
{% /dd %}

{% dd %}
**Description**: Connects the Worker to your secrets manager. See [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management/) for configuration information.
{% /dd %}

{% dt %}
`site`
{% /dt %}

{% dd %}
**Pipeline environment variable**: `DD_SITE`
{% /dd %}

{% dd %}
**Priority**: `DD_SITE`
{% /dd %}

{% dd %}
**Description**: Your Datadog site (optional, default: `datadoghq.com`).
{% /dd %}

{% dd %}
See [Getting Started with Sites](https://docs.datadoghq.com/getting_started/site/) for more information.
{% /dd %}

{% dt %}
`tags: []`
{% /dt %}

{% dd %}
**Pipeline environment variable**: `DD_OP_TAGS`
{% /dd %}

{% dd %}
**Priority**: `DD_OP_TAGS`
{% /dd %}

{% dd %}
**Description**: The tags reported with internal metrics and can be used to filter Observability Pipelines instances for Remote Configuration deployments.
{% /dd %}

{% dt %}
`threads`
{% /dt %}

{% dd %}
**Pipeline environment variable**: `DD_OP_THREADS`
{% /dd %}

{% dd %}
**Priority**: `DD_OP_THREADS`
{% /dd %}

{% dd %}
**Description**: The number of threads to use for processing (optional, default: the number of available cores).
{% /dd %}

{% /dl %}

## Other Worker configuration options{% #other-worker-configuration-options %}

Use the `VECTOR_HOSTNAME` environment variable to assign a unique hostname and help you identify the Worker.

## Enable the health check endpoint and the liveness and readiness probes{% #enable-the-health-check-endpoint-and-the-liveness-and-readiness-probes %}

Configure your load balancer's health check with the `/health` endpoint to check that the Worker is up and running.

For Kubernetes, the liveness and readiness probes are already enabled in the [helm chart](https://github.com/DataDog/helm-charts/blob/main/charts/observability-pipelines-worker/values.yaml#L33-L40) and [values.yaml](https://github.com/DataDog/helm-charts/blob/main/charts/observability-pipelines-worker/values.yaml#L303-L329) file.

For other installations such as VM-based ones, you must set `DD_OP_API_ENABLED` to `true` and set `DD_OP_API_ADDRESS` to `0.0.0.0:8686` to expose the `/health` endpoint. An example configuration:

```
api:
  enabled: true
  address: "0.0.0.0:8686"
```

## Further reading{% #further-reading %}

- [Redact data redaction with Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/sensitive_data_redaction/)
- [Update existing pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines/)