The Observability Pipelines Worker supports all major Kubernetes distributions, such as:

• Amazon Elastic Kubernetes Service (EKS)
• Azure Kubernetes Service (AKS)
• Google Kubernetes Engine (GKE)
• Red Hat OpenShift
• Rancher

This page describes how you can set up the Observability Pipelines Worker in ECS Fargate.

Configuration setup

In the following example, the setup configuration consists of a Fargate task, Fargate service, and a load balancer.

Configure the task definition

Create a task definition. The task definition describes which containers to run, the configuration (such as the environment variables and ports), and the CPU and memory resources allocated for the task.

The tasks should be deployed as a replica with auto scaling enabled, where the minimum number of containers should be based on your log volume and the maximum number of containers should be able to absorb any spikes or growth in log volume. See Best Practices for Scaling Observability Pipelines to help determine how much CPU and memory resources to allocate.

Notes:

• The guidance for CPU and memory allocation is not for a single instance of the task, but for the total number of tasks. For example, if you want to send 3 TB of logs to the Worker, you could either deploy three replicas with one vCPU each or deploy one replica with three vCPUs.
• Datadog recommends enabling load balancers for the pool of replica tasks.

In the task definition, set the DD_OP_SOURCE_* environment variable according to the configuration of the pipeline and port mappings. DD_OP_API_ENABLED and DD_OP_API_ADDRESS allow the load balancer to perform health checks on the Observability Pipelines Worker.

{
  "family": "my-opw",
  "containerDefinitions": [
    {
      "name": "my-opw",
      "image": "datadog/observability-pipelines-worker",
      "cpu": 0,
      "portMappings": [
        {
          "name": "my-opw-80-tcp",
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "command": [
        "run"
      ],
      "environment": [
        {
          "name": "DD_OP_API_ENABLED",
          "value": "true"
        },
        {
          "name": "DD_API_KEY",
          "value": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
        {
          "name": "DD_SITE",
          "value": "datadoghq.com"
        },
        {
          "name": "DD_OP_API_ADDRESS",
          "value": "0.0.0.0:8181"
        },
        {
          "name": "DD_OP_PIPELINE_ID",
          "value": "xxxxxxx-xxxx-xxxx-xxxx-xxxx"
        }
      ],
      "secrets": [
                {
                    "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:username_value",
                    "name": "username_value"
                }
            ],
      "mountPoints": [],
      "volumesFrom": [],
      "systemControls": []
    }
  ],
  "tags": [
    {
      "key": "PrincipalId",
      "value": "AROAYYB64AB3JW3TEST"
    },
    {
      "key": "User",
      "value": "username@test.com"
    }
  ],
  "executionRoleArn": "arn:aws:iam::60142xxxxxx:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "volumes": [],
  "placementConstraints": [],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "xxx",
  "memory": "xxx"
}

{
  "family": "my-opw",
  "containerDefinitions": [
    {
      "name": "my-opw",
      "image": "datadog/observability-pipelines-worker",
      "cpu": 0,
      "portMappings": [
        {
          "name": "my-opw-80-tcp",
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "command": [
        "run"
      ],
      "environment": [
        {
          "name": "DD_OP_API_ENABLED",
          "value": "true"
        },
        {
          "name": "DD_API_KEY",
          "value": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
        {
          "name": "DD_SITE",
          "value": "datadoghq.com"
        },
        {
          "name": "DD_OP_API_ADDRESS",
          "value": "0.0.0.0:8181"
        },
        {
          "name": "DD_OP_SOURCE_HTTP_SERVER_ADDRESS",
          "value": "0.0.0.0:80"
        },
        {
          "name": "DD_OP_PIPELINE_ID",
          "value": "xxxxxxx-xxxx-xxxx-xxxx-xxxx"
        }
      ],
      "mountPoints": [],
      "volumesFrom": [],
      "systemControls": []
    }
  ],
  "tags": [
    {
      "key": "PrincipalId",
      "value": "AROAYYB64AB3JW3TEST"
    },
    {
      "key": "User",
      "value": "username@test.com"
    }
  ],
  "executionRoleArn": "arn:aws:iam::60142xxxxxx:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "volumes": [],
  "placementConstraints": [],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "xxx",
  "memory": "xxx"
}

Configure the ECS service

Create an ECS service. The service configuration sets the number of Worker replicas to run and the scaling policy. In this example, the scaling policy is set to target an average CPU utilization of 70% with a minimum of two replicas and a maximum of five replicas.

Set up load balancing

Depending on your use case, configure either an Application Load Balancer or a Network Load Balancer to target the group of Fargate tasks you defined earlier. Configure the health check against the Observability Pipelines' API port that was set in the task definition.

Notes:

• By default, the Kubernetes Service maps incoming port <SERVICE_PORT> to the port the Worker is listening on (<TARGET_PORT>). If you want to map the Worker's pod port to a different incoming port of the Kubernetes Service, use the following service.ports[0].port and service.ports[0].targetPort values in the command:
  Copy

  --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282
  

• If you enable disk buffering for destinations, you must enable Kubernetes persistent volumes in the Observability Pipelines Helm chart.
• If you are using a firewall, see Add domains to firewall allowlist.

See Update Existing Pipelines if you want to make changes to your pipeline's configuration.

Self-hosted and self-managed Kubernetes clusters

If you are running a self-hosted and self-managed Kubernetes cluster, and defined zones with node labels using topology.kubernetes.io/zone, then you can use the Helm chart values file as is. However, if you are not using the label topology.kubernetes.io/zone, you need to update the topologyKey in the values.yaml file to match the key you are using. Or if you run your Kubernetes install without zones, remove the entire topology.kubernetes.io/zone section.

Kubernetes services created

When you install the Observability Pipelines Worker on Kubernetes, the Helm chart creates:

• A headless Service (clusterIP: None) that exposes the individual Worker Pods using DNS. This allows direct Pod-to-Pod communication and stable network identities for peer discovery or direct Pod addressing.
• A ClusterIP service that provides a single virtual IP and DNS name for the Worker. This enables load balancing across Worker Pods for internal cluster traffic.

LoadBalancer service

If you set service.type: LoadBalancer in the Helm chart, Kubernetes provisions a load balancer in supported environments and exposes the Worker Service with an external IP/DNS name. For example, Amazon EKS with the AWS Load Balancer Controller installed. Use this LoadBalancer service when traffic originates outside the cluster.

Set the Worker name using the Pod and cluster name

By default, a Worker's hostname is the machine's hostname, such as COMP-JLXPKWTGJF. If you run your pipeline across multiple clusters or containers, assign each Worker a unique hostname based on the Pod name and cluster name to make them easier to identify.

In the Helm chart values.yaml:

1. Configure the environment variable POD_NAME to be automatically set to the Pod's name. In the Helm chart:
   Copy

   env:
     - name: POD_NAME
       valueFrom:
        fieldRef:
           fieldPath: metadata.name
   

2. Set the CLUSTER_NAME environment variable in the Helm chart.
   Copy

   env:
     - name: CLUSTER_NAME
       value: "<MY_CLUSTER_NAME>"
   

3. Set the VECTOR_HOSTNAME to the POD_NAME and CLUSTER_NAME.
   Copy

   env:
     - name: POD_NAME
       valueFrom:
         fieldRef:
           fieldPath: metadata.name
   
     - name: CLUSTER_NAME
       value: "<MY_CLUSTER_NAME>"
   
     - name: VECTOR_HOSTNAME
       value: "$(CLUSTER_NAME)_$(POD_NAME)"
   

Upgrade the Worker

To upgrade the Worker, you can either upgrade with a pinned image tag or upgrade with the chart's default image tag.

After you update the values.yaml file with the new Worker version, new Worker pods are created with the updated image. Old pods are terminated only after the replacements are Ready. The update is a rolling update by default, so one pod is upgraded at a time. To change the default values, see the updateStrategy options in the values.yaml file:

updateStrategy: {}
#   type: RollingUpdate
#   rollingUpdate:
#     maxUnavailable: 1

# terminationGracePeriodSeconds -- Override terminationGracePeriodSeconds.

Upgrade with a pinned image tag

Update the image.tag in your values.yaml file, replacing <WORKER_VERSION> with the Worker version you want to use (such as 2.15.0):

image:
    # image.name -- Specify the image name to use (relative to `image.repository`).
    name: observability-pipelines-worker
    # image.tag -- Specify the image tag to use.
    tag: <WORKER_VERSION>

Note: Your deployment continues using that image tag until you set image.tag in your values.yaml file to another version.

Upgrade with the chart's default image tag

Run these commands to update the Helm repo and upgrade the release to the chart's default image, applying any overrides you previously set. Replace <YOUR_VALUES_FILE> with the name of your values.yaml file:

helm repo update
helm upgrade --install opw datadog/observability-pipelines-worker -f <YOUR_VALUES_FILE>.yaml

Note: Keep all custom settings in your values.yaml file and override only the values you need. You can then re-run helm upgrade without reapplying manual overrides.

Manually install the Worker

Follow these steps to manually install the Worker, instead of running the one-line installation script.

See Update Existing Pipelines if you want to make changes to your pipeline's configuration.

Upgrade the Worker

To upgrade the Worker to the latest version, run the following command:

sudo apt-get install --only-upgrade observability-pipelines-worker

sudo yum install --only-upgrade observability-pipelines-worker

Uninstall the Worker

If you want to uninstall the Worker, run the following commands:

sudo apt-get remove --purge observability-pipelines-worker

• api.
• config.
• http-intake.
• keys.
• install.
• yum.