--- title: Install the Worker description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > Observability Pipelines > Configuration > Install the Worker --- # Install the Worker {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} {% section displayed-if="Platform is Linux" %} This section only applies to users who meet the following criteria: Platform is Linux {% alert level="danger" %} For RHEL and CentOS, the Observability Pipelines Worker supports versions 8.0 or later. {% /alert %} {% /section %} The Observability Pipelines Worker is software that runs in your environment to centrally aggregate and process your logs and metrics (Preview (PREVIEW indicates an early access version of a major product or feature that you can opt into before its official release.)), and then route them to different destinations. {% section displayed-if="Platform is Kubernetes" %} This section only applies to users who meet the following criteria: Platform is Kubernetes The Observability Pipelines Worker supports all major Kubernetes distributions, such as: - Amazon Elastic Kubernetes Service (EKS) - Azure Kubernetes Service (AKS) - Google Kubernetes Engine (GKE) - Red Hat OpenShift - Rancher {% /section %} {% section displayed-if="Platform is ECS Fargate" %} This section only applies to users who meet the following criteria: Platform is ECS Fargate This page describes how you can set up the Observability Pipelines Worker in ECS Fargate. {% /section %} **Note**: If you are using a proxy, see the `proxy` option in [Bootstrap options](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md#bootstrap-options). ## Install the Worker{% #install-the-worker %} {% section displayed-if="The selected value for Pipeline setup method is included in the given list: 'API, Terraform'" %} This section only applies to users who meet the following criteria: The selected value for Pipeline setup method is included in the given list: 'API, Terraform' {% section displayed-if="Platform is Docker" %} This section only applies to users who meet the following criteria: Platform is Docker {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management 1. Run the following command to install the Worker: ``` docker run -i -e DD_API_KEY= \ -e DD_OP_PIPELINE_ID= \ -e DD_SITE= \ -v /path/to/local/bootstrap.yaml:/etc/observability-pipelines-worker/bootstrap.yaml \ datadog/observability-pipelines-worker run ``` You must replace the placeholders with the following values: - ``: Your Datadog API key. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). - ``: The ID of your pipeline. - ``: The [Datadog site](https://docs.datadoghq.com/getting_started/site.md). 1. Modify the Worker bootstrap file to connect the Worker to your secrets manager. See [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md) for more information. 1. Restart the Worker to use the updated bootstrap file: ``` sudo systemctl restart observability-pipelines-worker ``` {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables Run the following command to install the Worker: ``` docker run -i -e DD_API_KEY= \ -e DD_OP_PIPELINE_ID= \ -e DD_SITE= \ -e \ -e \ -p 8088:8088 \ datadog/observability-pipelines-worker run ``` You must replace the placeholders with these values: - ``: Your Datadog API key. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). - ``: The ID of your pipeline. - ``: The [Datadog site](https://docs.datadoghq.com/getting_started/site.md). - ``: The environment variables required by the source you are using for your pipeline. - For example: `DD_OP_SOURCE_DATADOG_AGENT_ADDRESS=0.0.0.0:8282` - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for a list of source environment variables. - ``: The environment variables required by the destinations you are using for your pipeline. - For example: `DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL=https://hec.splunkcloud.com:8088` - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for a list of destination environment variables. {% /section %} **Notes**: - By default, the `docker run` command exposes the same port the Worker is listening on. If you want to map the Worker's container port to a different port on the Docker host, use the `-p | --publish` option in the command: ``` -p 8282:8088 datadog/observability-pipelines-worker run ``` - Use the `VECTOR_HOSTNAME` environment variable to assign a unique hostname and help you identify the Worker. - If you are using a firewall, see Add domains to firewall allowlist. See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. {% /section %} {% section displayed-if="Platform is Kubernetes" %} This section only applies to users who meet the following criteria: Platform is Kubernetes 1. Download the [Helm chart values file](https://docs.datadoghq.com/resources/yaml/observability_pipelines/v2/setup/values.yaml.md). See the [full list of available configuration options](https://github.com/DataDog/helm-charts/blob/main/charts/observability-pipelines-worker/values.yaml). If you are not using a managed service, see Self-hosted and self-managed Kubernetes clusters before continuing to the next step. 1. Add the Datadog chart repository to Helm: ``` helm repo add datadog https://helm.datadoghq.com ``` If you already have the Datadog chart repository, run the following command to ensure it is up to date: ``` helm repo update ``` {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management See [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md?tab=kubernetes#configure-the-worker-to-retrieve-secrets) on how to configure your `values.yaml` file for your secrets manager. Run the following command to install the Worker: ``` helm upgrade --install opw \ -f values.yaml \ --set datadog.apiKey= \ --set datadog.pipelineId= \ datadog/observability-pipelines-worker ``` You must replace the placeholders with the following values: - ``: Your Datadog API key. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). - ``: The ID of your pipeline. {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables Run the following command to install the Worker: ``` helm upgrade --install opw \ -f values.yaml \ --set datadog.apiKey= \ --set datadog.pipelineId= \ --set \ --set \ --set service.ports[0].protocol=TCP,service.ports[0].port=,service.ports[0].targetPort= \ datadog/observability-pipelines-worker ``` You must replace the placeholders with the following values: - ``: Your Datadog API key. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). - ``: The ID of your pipeline. - ``: The environment variables required by the source you are using for your pipeline. - For example: `--set env[0].name=DD_OP_SOURCE_DATADOG_AGENT_ADDRESS,env[0].value='0.0.0.0' \` - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for a list of source environment variables. - ``: The environment variables required by the destinations you are using for your pipeline. - For example: `--set env[1].name=DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL,env[1].value='https://hec.splunkcloud.com:8088' \` - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for a list of destination environment variables. {% /section %} {% /section %} {% section displayed-if="Platform is Linux" %} This section only applies to users who meet the following criteria: Platform is Linux Follow these steps if you want to use the one-line installation script to install the Worker. Otherwise, see Manually install the Worker. {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management 1. Run this one-step command to install the Worker: ``` DD_API_KEY= DD_OP_PIPELINE_ID= DD_SITE= bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_op_worker2.sh)" ``` You must replace the placeholders with the following values: - ``: Your Datadog API key. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). - ``: The ID of your pipeline. - ``: The [Datadog site](https://docs.datadoghq.com/getting_started/site.md). 1. Modify the Worker bootstrap file to connect the Worker to your secrets manager. See [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md) for more information. 1. Restart the Worker to use the updated bootstrap file: ``` sudo systemctl restart observability-pipelines-worker ``` **Notes**: - The environment variables used by the Worker in `/etc/default/observability-pipelines-worker` are not updated on subsequent runs of the install script. To update environment variables, manually make the changes in the file and restart the Worker. - See Add domains to firewall allowlist if you are using a firewall. See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables Run this one-step command to install the Worker: ``` DD_API_KEY= DD_OP_PIPELINE_ID= DD_SITE= bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_op_worker2.sh)" ``` You must replace the placeholders with the following values: - ``: Your Datadog API key. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). - ``: The ID of your pipeline. - ``: The [Datadog site](https://docs.datadoghq.com/getting_started/site.md). - ``: The environment variables required by the source you are using for your pipeline. - For example: `DD_OP_SOURCE_DATADOG_AGENT_ADDRESS=0.0.0.0:8282` - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for a list of source environment variables. - ``: The environment variables required by the destinations you are using for your pipeline. - For example: `DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL=https://hec.splunkcloud.com:8088` - See [Environment Variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for a list of destination environment variables. **Notes**: - The environment variables used by the Worker in `/etc/default/observability-pipelines-worker` are not updated on subsequent runs of the install script. To update environment variables, manually make the changes in the file and restart the Worker. - See Add domains to firewall allowlist if you are using a firewall. See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. {% /section %} {% /section %} {% section displayed-if="Platform is CloudFormation" %} This section only applies to users who meet the following criteria: Platform is CloudFormation 1. Select one of the options in the dropdown to provide the expected log or metrics (Preview (PREVIEW indicates an early access version of a major product or feature that you can opt into before its official release.)) volume for the pipeline: | Option | Description | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Unsure | Use this option if you are not able to project the data volume or you want to test the Worker. This option provisions the EC2 Auto Scaling group with a maximum of 2 general purpose `t4g.large` instances. | | 1-5 TB/day | This option provisions the EC2 Auto Scaling group with a maximum of 2 compute optimized instances `c6g.large`. | | 5-10 TB/day | This option provisions the EC2 Auto Scaling group with a minimum of 2 and a maximum of 5 compute optimized `c6g.large` instances. | | \>10 TB/day | Datadog recommends this option for large-scale production deployments. It provisions the EC2 Auto Scaling group with a minimum of 2 and a maximum of 10 compute optimized `c6g.xlarge` instances. | **Note**: All other parameters are set to reasonable defaults for a Worker deployment, but you can adjust them for your use case as needed in the AWS Console before creating the stack. 1. Select the AWS region you want to use to install the Worker. 1. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). 1. Click **Launch CloudFormation Template** to navigate to the AWS Console to review the stack configuration and then launch it. Make sure the CloudFormation parameters are as expected. 1. Select the VPC and subnet you want to use to install the Worker. 1. Review and check the necessary permissions checkboxes for IAM. Click **Submit** to create the stack. CloudFormation handles the installation at this point; the Worker instances are launched, the necessary software is downloaded, and the Worker starts automatically. **Note**: If you are using a firewall, see Add domains to firewall allowlist. See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. {% /section %} {% /section %} {% section displayed-if="Pipeline setup method is UI" %} This section only applies to users who meet the following criteria: Pipeline setup method is UI {% section displayed-if="Platform is Docker" %} This section only applies to users who meet the following criteria: Platform is Docker {% image source="https://docs.dd-static.net/images/observability_pipelines/install_page_secrets.c6bce4438eaad5a7614fd3c9e634e740.png?auto=format" alt="The install page in the UI with a dropdown menu to choose your installation platform and fields to enter environment variables." /%} After you set up your source, destinations, and processors on the Build page of the pipeline UI, follow the steps on the Install page to install the Worker. 1. Select **Docker** as your installation platform. {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management In **Review your secrets management**, ensure that your secrets are configured in your secrets manager. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). Run the command provided in the UI to install the Worker. The command points to the Worker bootstrap file that you configure to resolve secrets using your secrets manager. Modify the Worker bootstrap file to connect the Worker to your secrets manager. See [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md) for more information. Restart the Worker to use the updated bootstrap file: ``` sudo systemctl restart observability-pipelines-worker ``` Navigate back to the Observability Pipelines installation page and click **Deploy**. {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables In **Review your secrets management**, enter the [environment variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for your sources and destinations, if applicable. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). Run the command provided in the UI to install the Worker. The command is automatically populated with the environment variables you entered in **Review your secrets management**. Navigate back to the Observability Pipelines installation page and click **Deploy**. {% /section %} **Notes**: - By default, the `docker run` command exposes the same port the Worker is listening on. If you want to map the Worker's container port to a different port on the Docker host, use the `-p | --publish` option in the command: ``` -p 8282:8088 datadog/observability-pipelines-worker run ``` - Use the `VECTOR_HOSTNAME` environment variable to assign a unique hostname and help you identify the Worker. - If you are using a firewall, see Add domains to firewall allowlist. See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. {% /section %} {% section displayed-if="Platform is Kubernetes" %} This section only applies to users who meet the following criteria: Platform is Kubernetes {% image source="https://docs.dd-static.net/images/observability_pipelines/install_page_secrets.c6bce4438eaad5a7614fd3c9e634e740.png?auto=format" alt="The install page in the UI with a dropdown menu to choose your installation platform and fields to enter environment variables." /%} After you set up your source, destinations, and processors on the Build page of the pipeline UI, follow the steps on the Install page to install the Worker. 1. Select **Kubernetes** as your installation platform. {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management In **Review your secrets management**, ensure that your secrets are configured in your secrets manager.Download the [Helm chart values file](https://docs.datadoghq.com/resources/yaml/observability_pipelines/v2/setup/values.yaml.md). See the [full list of available configuration options](https://github.com/DataDog/helm-charts/blob/main/charts/observability-pipelines-worker/values.yaml). - If you are not using a managed service, see Self-hosted and self-managed Kubernetes clusters before continuing to the next step. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). Add the Datadog chart repository to Helm: ``` helm repo add datadog https://helm.datadoghq.com ``` If you already have the Datadog chart repository, run the following command to make sure it is up to date: ``` helm repo update ``` Configure your `values.yaml` file for your secrets manager. See [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md?tab=kubernetes#configure-the-worker-to-retrieve-secrets).Run the command provided in the UI to install the Worker.Navigate back to the Observability Pipelines installation page and click **Deploy**.{% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables In **Review your secrets management**, enter the [environment variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for your sources and destinations, if applicable.Download the [Helm chart values file](https://docs.datadoghq.com/resources/yaml/observability_pipelines/v2/setup/values.yaml.md). See the [full list of available configuration options](https://github.com/DataDog/helm-charts/blob/main/charts/observability-pipelines-worker/values.yaml). - If you are not using a managed service, see Self-hosted and self-managed Kubernetes clusters before continuing to the next step. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). Add the Datadog chart repository to Helm: ``` helm repo add datadog https://helm.datadoghq.com ``` If you already have the Datadog chart repository, run the following command to make sure it is up to date: ``` helm repo update ``` Run the command provided in the UI to install the Worker. The command is automatically populated with the environment variables you entered earlier.Navigate back to the Observability Pipelines installation page and click **Deploy**.{% /section %} {% /section %} {% section displayed-if="Platform is Linux" %} This section only applies to users who meet the following criteria: Platform is Linux {% image source="https://docs.dd-static.net/images/observability_pipelines/install_page_secrets.c6bce4438eaad5a7614fd3c9e634e740.png?auto=format" alt="The install page in the UI with a dropdown menu to choose your installation platform and fields to enter environment variables." /%} After you set up your source, destinations, and processors on the Build page of the pipeline UI, follow the steps on the Install page to install the Worker. 1. Select **Linux** as your installation platform. {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management In **Review your secrets management**, ensure that your secrets are configured in your secrets manager. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). Run the one-step command provided in the UI to install the Worker. If you want to manually install the Worker, see Manually install the Worker. Modify the Worker bootstrap file to connect the Worker to your secrets manager. See [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md) for more information.Restart the Worker to use the updated bootstrap file: ``` sudo systemctl restart observability-pipelines-worker ``` Navigate back to the Observability Pipelines installation page and click **Deploy**. **Note**: If you are using a firewall, see Add domains to firewall allowlist. {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables In **Review your secrets management**, enter the [environment variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for your sources and destinations, if applicable. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). Run the one-step command provided in the UI to install the Worker. If you want to manually install the Worker, see Manually install the Worker. Navigate back to the Observability Pipelines installation page and click **Deploy**. **Notes**: - The environment variables used by the Worker in `/etc/default/observability-pipelines-worker` are not updated on subsequent runs of the install script. To update environment variables, manually make the changes in the file and restart the Worker. - See Add domains to firewall allowlist if you are using a firewall. {% /section %} See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. {% /section %} {% section displayed-if="Platform is CloudFormation" %} This section only applies to users who meet the following criteria: Platform is CloudFormation {% image source="https://docs.dd-static.net/images/observability_pipelines/install_page_secrets.c6bce4438eaad5a7614fd3c9e634e740.png?auto=format" alt="The install page in the UI with a dropdown menu to choose your installation platform and fields to enter environment variables." /%} After you set up your source, destinations, and processors on the Build page of the pipeline UI, follow the steps on the Install page to install the Worker. 1. Select **CloudFormation** as your installation platform. {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management In **Review your secrets management**, ensure that your secrets are configured in your secrets manager.{% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables In **Review your secrets management**, enter the [environment variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) for your sources and destinations, if applicable.{% /section %} Select one of the options in the dropdown to provide the expected log or metrics (Preview (PREVIEW indicates an early access version of a major product or feature that you can opt into before its official release.)) volume for the pipeline: | Option | Description | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Unsure | Use this option if you are not able to project the data volume or you want to test the Worker. This option provisions the EC2 Auto Scaling group with a maximum of 2 general purpose `t4g.large` instances. | | 1-5 TB/day | This option provisions the EC2 Auto Scaling group with a maximum of 2 compute optimized instances `c6g.large`. | | 5-10 TB/day | This option provisions the EC2 Auto Scaling group with a minimum of 2 and a maximum of 5 compute optimized `c6g.large` instances. | | \>10 TB/day | Datadog recommends this option for large-scale production deployments. It provisions the EC2 Auto Scaling group with a minimum of 2 and a maximum of 10 compute optimized `c6g.xlarge` instances. | **Note**: All other parameters are set to reasonable defaults for a Worker deployment, but you can adjust them for your use case as needed in the AWS Console before creating the stack. Select the AWS region you want to use to install the Worker. Click **Select API key** to choose the Datadog API key you want to use. - **Note**: The API key must be [enabled for Remote Configuration](https://app.datadoghq.com/organization-settings/remote-config/setup). Click **Launch CloudFormation Template** to navigate to the AWS Console to review the stack configuration and then launch it. Make sure the CloudFormation parameters are as expected. Select the VPC and subnet you want to use to install the Worker. Review and check the necessary permissions checkboxes for IAM. Click **Submit** to create the stack. CloudFormation handles the installation at this point; the Worker instances are launched, the necessary software is downloaded, and the Worker starts automatically. Navigate back to the Observability Pipelines installation page and click **Deploy**. **Note**: If you are using a firewall, see Add domains to firewall allowlist. See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. {% /section %} {% /section %} {% section displayed-if="Platform is ECS Fargate" %} This section only applies to users who meet the following criteria: Platform is ECS Fargate ### Configuration setup{% #configuration-setup %} In the following example, the setup configuration consists of a Fargate task, Fargate service, and a load balancer. {% image source="https://docs.dd-static.net/images/observability_pipelines/worker_fargate_architecture.a5b2ebc6ad385bb1535541ab4967877d.png?auto=format" alt="An architecture diagram with logs going to an application load balancer, a OP Worker task, and the Fargate service" /%} ### Configure the task definition{% #configure-the-task-definition %} [Create a task definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html). The task definition describes which containers to run, the configuration (such as the environment variables and ports), and the CPU and memory resources allocated for the task. The tasks should be deployed as a replica with auto scaling enabled, where the minimum number of containers should be based on your log volume and the maximum number of containers should be able to absorb any spikes or growth in log volume. See [Best Practices for Scaling Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/scaling_and_performance/best_practices_for_scaling_observability_pipelines.md) to help determine how much CPU and memory resources to allocate. **Notes**: - The guidance for CPU and memory allocation is not for a single instance of the task, but for the total number of tasks. For example, if you want to send 3 TB of logs to the Worker, you could either deploy three replicas with one vCPU each or deploy one replica with three vCPUs. - Datadog recommends enabling load balancers for the pool of replica tasks. In the task definition, set the `DD_OP_SOURCE_*` environment variable according to the configuration of the pipeline and port mappings. `DD_OP_API_ENABLED` and `DD_OP_API_ADDRESS` allow the load balancer to perform health checks on the Observability Pipelines Worker. {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management Also, configure your task definition to point to your [Secrets Manager](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md?tab=kubernetes#configure-the-worker-to-retrieve-secrets). For example for AWS Secrets Manager or AWS Systems Manager Parameter Store, specify the ARNs that point to the stored secrets. See [Specifying sensitive data using Secrets Manager secrets in Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-tutorial.html) for details. An example task definition using AWS Secrets Manager: ``` { "family": "my-opw", "containerDefinitions": [ { "name": "my-opw", "image": "datadog/observability-pipelines-worker", "cpu": 0, "portMappings": [ { "name": "my-opw-80-tcp", "containerPort": 80, "hostPort": 80, "protocol": "tcp" } ], "essential": true, "command": [ "run" ], "environment": [ { "name": "DD_OP_API_ENABLED", "value": "true" }, { "name": "DD_API_KEY", "value": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, { "name": "DD_SITE", "value": "datadoghq.com" }, { "name": "DD_OP_API_ADDRESS", "value": "0.0.0.0:8181" }, { "name": "DD_OP_PIPELINE_ID", "value": "xxxxxxx-xxxx-xxxx-xxxx-xxxx" } ], "secrets": [ { "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:username_value", "name": "username_value" } ], "mountPoints": [], "volumesFrom": [], "systemControls": [] } ], "tags": [ { "key": "PrincipalId", "value": "AROAYYB64AB3JW3TEST" }, { "key": "User", "value": "username@test.com" } ], "executionRoleArn": "arn:aws:iam::60142xxxxxx:role/ecsTaskExecutionRole", "networkMode": "awsvpc", "volumes": [], "placementConstraints": [], "requiresCompatibilities": [ "FARGATE" ], "cpu": "xxx", "memory": "xxx" } ``` {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables An example task definition: ``` { "family": "my-opw", "containerDefinitions": [ { "name": "my-opw", "image": "datadog/observability-pipelines-worker", "cpu": 0, "portMappings": [ { "name": "my-opw-80-tcp", "containerPort": 80, "hostPort": 80, "protocol": "tcp" } ], "essential": true, "command": [ "run" ], "environment": [ { "name": "DD_OP_API_ENABLED", "value": "true" }, { "name": "DD_API_KEY", "value": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" }, { "name": "DD_SITE", "value": "datadoghq.com" }, { "name": "DD_OP_API_ADDRESS", "value": "0.0.0.0:8181" }, { "name": "DD_OP_SOURCE_HTTP_SERVER_ADDRESS", "value": "0.0.0.0:80" }, { "name": "DD_OP_PIPELINE_ID", "value": "xxxxxxx-xxxx-xxxx-xxxx-xxxx" } ], "mountPoints": [], "volumesFrom": [], "systemControls": [] } ], "tags": [ { "key": "PrincipalId", "value": "AROAYYB64AB3JW3TEST" }, { "key": "User", "value": "username@test.com" } ], "executionRoleArn": "arn:aws:iam::60142xxxxxx:role/ecsTaskExecutionRole", "networkMode": "awsvpc", "volumes": [], "placementConstraints": [], "requiresCompatibilities": [ "FARGATE" ], "cpu": "xxx", "memory": "xxx" } ``` {% /section %} ### Configure the ECS service{% #configure-the-ecs-service %} [Create an ECS service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-console-v2.html). The service configuration sets the number of Worker replicas to run and the scaling policy. In this example, the scaling policy is set to target an average CPU utilization of 70% with a minimum of two replicas and a maximum of five replicas. ### Set up load balancing{% #set-up-load-balancing %} Depending on your use case, configure either an [Application Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html) or a [Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html) to target the group of Fargate tasks you defined earlier. Configure the health check against the Observability Pipelines' API port that was set in the task definition. {% /section %} {% section displayed-if="Platform is Kubernetes" %} This section only applies to users who meet the following criteria: Platform is Kubernetes **Notes**: - By default, the Kubernetes Service maps incoming port `` to the port the Worker is listening on (``). If you want to map the Worker's pod port to a different incoming port of the Kubernetes Service, use the following `service.ports[0].port` and `service.ports[0].targetPort` values in the command: ``` --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282 ``` - If you enable [disk buffering](https://docs.datadoghq.com/observability_pipelines/scaling_and_performance/buffering_and_backpressure.md#destination-buffers) for destinations, you must enable Kubernetes [persistent volumes](https://github.com/DataDog/helm-charts/blob/main/charts/observability-pipelines-worker/values.yaml#L278) in the Observability Pipelines Helm chart. - If you are using a firewall, see Add domains to firewall allowlist. See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. ### Self-hosted and self-managed Kubernetes clusters{% #self-hosted-and-self-managed-kubernetes-clusters %} If you are running a self-hosted and self-managed Kubernetes cluster, and defined zones with node labels using `topology.kubernetes.io/zone`, then you can use the Helm chart values file as is. However, if you are not using the label `topology.kubernetes.io/zone`, you need to update the `topologyKey` in the `values.yaml` file to match the key you are using. Or if you run your Kubernetes install without zones, remove the entire `topology.kubernetes.io/zone` section. ### Kubernetes services created{% #kubernetes-services-created %} When you install the Observability Pipelines Worker on Kubernetes, the Helm chart creates: - A headless Service (`clusterIP: None`) that exposes the individual Worker Pods using DNS. This allows direct Pod-to-Pod communication and stable network identities for peer discovery or direct Pod addressing. - A ClusterIP service that provides a single virtual IP and DNS name for the Worker. This enables load balancing across Worker Pods for internal cluster traffic. ### LoadBalancer service{% #loadbalancer-service %} If you set `service.type: LoadBalancer` in the Helm chart, Kubernetes provisions a load balancer in supported environments and exposes the Worker Service with an external IP/DNS name. For example, Amazon EKS with the [AWS Load Balancer Controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) installed. Use this `LoadBalancer` service when traffic originates outside the cluster. ### Set the Worker name using the Pod and cluster name{% #set-the-worker-name-using-the-pod-and-cluster-name %} By default, a Worker's hostname is the machine's hostname, such as `COMP-JLXPKWTGJF`. If you run your pipeline across multiple clusters or containers, assign each Worker a unique hostname based on the Pod name and cluster name to make them easier to identify. In the Helm chart [`values.yaml`](https://github.com/DataDog/helm-charts/blob/main/charts/observability-pipelines-worker/values.yaml): 1. Configure the environment variable `POD_NAME` to be automatically set to the Pod's name. In the Helm chart: ``` env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name ``` 1. Set the `CLUSTER_NAME` environment variable in the Helm chart. ``` env: - name: CLUSTER_NAME value: "" ``` 1. Set the `VECTOR_HOSTNAME` to the `POD_NAME` and `CLUSTER_NAME`. ``` env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: CLUSTER_NAME value: "" - name: VECTOR_HOSTNAME value: "$(CLUSTER_NAME)_$(POD_NAME)" ``` ## Upgrade the Worker{% #upgrade-the-worker %} To upgrade the Worker, you can either upgrade with a pinned image tag or upgrade with the chart's default image tag. After you update the `values.yaml` file with the new Worker version, new Worker pods are created with the updated image. Old pods are terminated only after the replacements are `Ready`. The update is a rolling update by default, so one pod is upgraded at a time. To change the default values, see the `updateStrategy` options in the `values.yaml` file: ``` updateStrategy: {} # type: RollingUpdate # rollingUpdate: # maxUnavailable: 1 # terminationGracePeriodSeconds -- Override terminationGracePeriodSeconds. ``` ### Upgrade with a pinned image tag{% #upgrade-with-a-pinned-image-tag %} Update the `image.tag` in your `values.yaml` file, replacing `` with the Worker version you want to use (such as `2.15.0`): ``` image: # image.name -- Specify the image name to use (relative to `image.repository`). name: observability-pipelines-worker # image.tag -- Specify the image tag to use. tag: ``` **Note**: Your deployment continues using that image tag until you set `image.tag` in your `values.yaml` file to another version. ### Upgrade with the chart's default image tag{% #upgrade-with-the-chart-s-default-image-tag %} Run these commands to update the Helm repo and upgrade the release to the chart's default image, applying any overrides you previously set. Replace `` with the name of your `values.yaml` file: ``` helm repo update helm upgrade --install opw datadog/observability-pipelines-worker -f .yaml ``` **Note**: Keep all custom settings in your `values.yaml` file and override only the values you need. You can then re-run `helm upgrade` without reapplying manual overrides. {% /section %} {% section displayed-if="Platform is Linux" %} This section only applies to users who meet the following criteria: Platform is Linux ### Manually install the Worker{% #manually-install-the-worker %} Follow these steps to manually install the Worker, instead of running the one-line installation script. {% tab title="APT" %} 1. Set up APT transport for downloading using HTTPS: ``` sudo apt-get update sudo apt-get install apt-transport-https curl gnupg ``` 1. Run the following commands to set up the Datadog `deb` repo on your system and create a Datadog archive keyring: ``` sudo sh -c "echo 'deb [signed-by=/usr/share/keyrings/datadog-archive-keyring.gpg] https://apt.datadoghq.com/ stable observability-pipelines-worker-2' > /etc/apt/sources.list.d/datadog-observability-pipelines-worker.list" sudo touch /usr/share/keyrings/datadog-archive-keyring.gpg sudo chmod a+r /usr/share/keyrings/datadog-archive-keyring.gpg curl https://keys.datadoghq.com/DATADOG_APT_KEY_CURRENT.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch curl https://keys.datadoghq.com/DATADOG_APT_KEY_06462314.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch curl https://keys.datadoghq.com/DATADOG_APT_KEY_F14F620E.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch curl https://keys.datadoghq.com/DATADOG_APT_KEY_C0962C7D.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch ``` 1. Run the following commands to update your local `apt` repo and install the Worker: ``` sudo apt-get update sudo apt-get install observability-pipelines-worker datadog-signing-keys ``` {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management Add your Datadog API key, site (for example, `datadoghq.com` for US1), and pipeline ID to the Worker's environment file: ``` sudo cat < /etc/default/observability-pipelines-worker DD_API_KEY= DD_OP_PIPELINE_ID= DD_SITE= EOF ``` {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables Add your Datadog API key, site (for example, `datadoghq.com` for US1), and source and destination [environment variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) to the Worker's environment file: ``` sudo cat < /etc/default/observability-pipelines-worker DD_API_KEY= DD_OP_PIPELINE_ID= DD_SITE= EOF ``` {% /section %} Start the Worker: ``` sudo systemctl restart observability-pipelines-worker ``` {% section displayed-if="Pipeline setup method is UI" %} This section only applies to users who meet the following criteria: Pipeline setup method is UI Navigate back to the Observability Pipelines installation page and click **Deploy**. **Notes**: - The environment variables used by the Worker in `/etc/default/observability-pipelines-worker` are not updated on subsequent runs of the install script. To update environment variables, manually make the changes in the file and restart the Worker. - See Add domains to firewall allowlist if you are using a firewall. {% /section %} {% /tab %} {% tab title="RPM" %} 1. Set up the Datadog `rpm` repo on your system with the following command. **Note**: If you are running RHEL 8.1 or CentOS 8.1, use `repo_gpgcheck=0` instead of `repo_gpgcheck=1` in the configuration below. ``` cat < /etc/yum.repos.d/datadog-observability-pipelines-worker.repo [observability-pipelines-worker] name = Observability Pipelines Worker baseurl = https://yum.datadoghq.com/stable/observability-pipelines-worker-2/\$basearch/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://keys.datadoghq.com/DATADOG_RPM_KEY_CURRENT.public https://keys.datadoghq.com/DATADOG_RPM_KEY_B01082D3.public EOF ``` 1. Update your packages and install the Worker: ``` sudo yum makecache sudo yum install observability-pipelines-worker ``` {% section displayed-if="Secrets retrieval method is Secrets Management" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Secrets Management Add your Datadog API key, site (for example, `datadoghq.com` for US1), and pipeline ID to the Worker's environment file: ``` sudo cat <<-EOF > /etc/default/observability-pipelines-worker DD_API_KEY= DD_OP_PIPELINE_ID= DD_SITE= EOF ``` Start the Worker: ``` sudo systemctl restart observability-pipelines-worker ``` {% /section %} {% section displayed-if="Secrets retrieval method is Environment Variables" %} This section only applies to users who meet the following criteria: Secrets retrieval method is Environment Variables Add your Datadog API key, site (for example, `datadoghq.com` for US1), and source and destination [environment variables](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) to the Worker's environment file: ``` sudo cat <<-EOF > /etc/default/observability-pipelines-worker DD_API_KEY= DD_OP_PIPELINE_ID= DD_SITE= EOF ``` Start the Worker: ``` sudo systemctl restart observability-pipelines-worker ``` {% /section %} {% section displayed-if="Pipeline setup method is UI" %} This section only applies to users who meet the following criteria: Pipeline setup method is UI Navigate back to the Observability Pipelines installation page and click **Deploy**. **Notes**: - The environment variables used by the Worker in `/etc/default/observability-pipelines-worker` are not updated on subsequent runs of the install script. To update environment variables, manually make the changes in the file and restart the Worker. - See Add domains to firewall allowlist if you are using a firewall. {% /section %} {% /tab %} {% section displayed-if="The selected value for Pipeline setup method is included in the given list: 'API, Terraform'" %} This section only applies to users who meet the following criteria: The selected value for Pipeline setup method is included in the given list: 'API, Terraform' **Notes**: - The environment variables used by the Worker in `/etc/default/observability-pipelines-worker` are not updated on subsequent runs of the install script. To update environment variables, manually make the changes in the file and restart the Worker. - See Add domains to firewall allowlist if you are using a firewall. {% /section %} See [Update Existing Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/update_existing_pipelines.md) if you want to make changes to your pipeline's configuration. ## Upgrade the Worker{% #upgrade-the-worker-2 %} To upgrade the Worker to the latest version, run the following command: {% tab title="APT" %} ``` sudo apt-get install --only-upgrade observability-pipelines-worker ``` {% /tab %} {% tab title="RPM" %} ``` sudo yum install --only-upgrade observability-pipelines-worker ``` {% /tab %} ## Uninstall the Worker{% #uninstall-the-worker %} If you want to uninstall the Worker, run the following commands: {% tab title="APT" %} ``` sudo apt-get remove --purge observability-pipelines-worker ``` {% /tab %} {% tab title="RPM" %} 1. ``` yum remove observability-pipelines-worker ``` 1. ``` rpm -q --configfiles observability-pipelines-worker ``` {% /tab %} {% /section %} ## Add domains to firewall allowlist{% #add-domains-to-firewall-allowlist %} If you are using a firewall, these domains must be added to the allowlist: {% section displayed-if="The selected value for Platform is included in the given list: 'Docker, Kubernetes, CloudFormation, ECS Fargate'" %} This section only applies to users who meet the following criteria: The selected value for Platform is included in the given list: 'Docker, Kubernetes, CloudFormation, ECS Fargate' - `api.` - `config.` - `http-intake.` - `keys.` {% /section %} {% section displayed-if="Platform is Linux" %} This section only applies to users who meet the following criteria: Platform is Linux - `api.` - `config.` - `http-intake.` - `keys.` - `install.` - `yum.` {% /section %} ## Index your Worker logs{% #index-your-worker-logs %} Make sure your Worker logs are [indexed](https://docs.datadoghq.com/logs/log_configuration/indexes.md) in Log Management for optimal functionality. The logs provide deployment information, such as Worker status, version, and any errors, that is shown in the UI. The logs are also helpful for troubleshooting Worker or pipelines issues. All Worker logs have the tag `source:op_worker`. ## Further reading{% #further-reading %} - [Advanced Worker configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md) - [Learn more about Secrets Management in Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md) - [Worker CLI commands](https://docs.datadoghq.com/observability_pipelines/monitoring_and_troubleshooting/worker_cli_commands.md) - [Environment variable for sources, processors, and components](https://docs.datadoghq.com/observability_pipelines/guide/environment_variables.md) - [Set up pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md) {% image source="https://docs.dd-static.net/images/observability_pipelines/install_page_secrets.c6bce4438eaad5a7614fd3c9e634e740.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/install_page_secrets.c6bce4438eaad5a7614fd3c9e634e740.png?auto=format&fit=max&w=850&dpr=2 2x" alt="" /%} {% image source="https://docs.dd-static.net/images/observability_pipelines/worker_fargate_architecture.a5b2ebc6ad385bb1535541ab4967877d.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/observability_pipelines/worker_fargate_architecture.a5b2ebc6ad385bb1535541ab4967877d.png?auto=format&fit=max&w=850&dpr=2 2x" alt="" /%}