---
title: Upgrade the Worker Guide
description: >-
  Learn about new features, enhancements, and fixes for Worker versions 2.7 to
  2.25.
breadcrumbs: >-
  Docs > Observability Pipelines > Observability Pipelines Guides > Upgrade the
  Worker Guide
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Upgrade the Worker Guide

## Overview{% #overview %}

{% alert level="info" %}
Datadog recommends updating the Observability Pipelines Worker (OPW) with every minor and patch release, or monthly at a minimum.  Upgrading to the latest major OPW version and keeping it updated is the only supported way to get new OPW functionalities, fixes, and security updates.
{% /alert %}

This guide goes over how to upgrade to a specific Worker version and the updates for that version.

## Worker version 2.25.1{% #worker-version-2251 %}

To upgrade to Worker version 2.25.1:

- Docker: Run the `docker pull` command for the [2.25.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.25.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.25.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.25.1`.

Worker version 2.25.1 gives you access to the following:

#### Security{% #security %}

- The Microsoft Sentinel destination now verifies server certificates and hostnames for Azure OAuth token requests and log ingestion requests.

#### Fixes{% #fixes %}

- The Microsoft Sentinel destination now:
  - Splits batches into multiple requests as needed, so each request stays under the Azure 1 MB request size limit.
  - Refreshes its Azure OAuth token 60 seconds before the token expires. If a refresh fails, it logs an error and retries every 2 seconds until a refresh succeeds, then returns to that schedule.
  - Times out Azure OAuth token requests after 30 seconds and retries them like any other failed refresh.
- The `rc_version` tag on pipeline component metrics and the `op_component.rc_version` field in component logs now report the latest configuration version that each component runs, including components that a pipeline update didn't change.

## Worker version 2.25.0{% #worker-version-2250 %}

To upgrade to Worker version 2.25.0:

- Docker: Run the `docker pull` command for the [2.25.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.25.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.25.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.25.0`.

Worker version 2.25.0 gives you access to the following:

#### Security{% #security-1 %}

- Updated the Worker's bundled Mozilla root certificate store. The store no longer includes `emSign ECC Root CA - G3`, which was removed after the [eMudhra intermediate certificate compromise](https://trust.datadoghq.com/?tcuUid=301abdf4-0ad2-4db0-9843-c447a0714bfe), or other roots that Mozilla has distrusted. As a result:
  - Components that use the built-in root store no longer trust certificate chains through the distrusted emSign intermediates.
  - TLS connections to servers whose certificate chains end at a removed root fail validation.
- Error messages that the Worker reports to Datadog when a component crashes now go through the same sensitive data redaction rules as the Worker's logs.

#### Enhancements{% #enhancements %}

- The Grok Parser processor's logs now include:

  - A `rule` field that identifies the rule's source.
  - A `rule_kind` field that indicates the rule's type as `custom` or `library`.

Rule compilation error logs no longer include a separate `source` field. If you filter on `source`, filter on `rule` instead.

- The Datadog BYOC Logs destination now presents a rotated mTLS client certificate on the next request after it detects the rotation. Previously, the destination kept using the old certificate until existing connections reached their maximum age, which could take up to an hour. Connections are still re-established at least once an hour.

- The `DD_OP_REDUCE_WINDOW_SECS` environment variable is now available for setting how long Reduce processors aggregate events before they emit a group.

- The Sample processor now limits memory for tracking sampling groups to 128 MiB. When it reaches the limit, it evicts the least-recently used groups.

- The Worker has improved log ingestion performance.

#### Fixes{% #fixes-1 %}

- When a source or destination fails, the Worker now logs the underlying error, such as `Address already in use`, instead of only `the task completed with an error`. If a component panics, the logs also include the panic message.
- The Google Pub/Sub source now reconnects immediately, without logging an error, when the server resets a streaming pull.
- The Google Pub/Sub source now trusts every certificate in the configured CA file, not only the last one.
- The OpenTelemetry source now responds to gRPC requests for unsupported services with the gRPC status `UNIMPLEMENTED` instead of HTTP 404. These requests no longer log an error or increment the `component_errors_total` metric.
- The Worker now tracks internal logs dropped under high load in `component_discarded_events_total` with `intentional:false`. This affects only internal components (`component_id:_*`), not pipeline data.

## Worker version 2.24.1{% #worker-version-2241 %}

To upgrade to Worker version 2.24.1:

- Docker: Run the `docker pull` command for the [2.24.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.24.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.24.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.24.1`.

Worker version 2.24.1 gives you access to the following:

#### Security{% #security-2 %}

- Updated `rustls-webpki` to version 0.103.16. The Worker now rejects certificate chains that include any of the [eMudhra intermediate certificate authorities (CAs) whose private keys were reported as compromised](https://trust.datadoghq.com/?tcuUid=301abdf4-0ad2-4db0-9843-c447a0714bfe). The Worker rejects these chains even when the issuing root CA is trusted.

This protection applies only to components that use rustls. The following components use the system CA trust store instead:

  - Components that use OpenSSL
  - The Amazon Security Lake destination's AWS account lookup
  - The Google Cloud Pub/Sub source

To protect components that use the system CA trust store:

  - **Container image**: No action is needed unless you mount your own CA bundle or set `SSL_CERT_FILE`. The Worker container image distrusts `emSign ECC Root CA - G3`.
  - **APT and RPM packages**: Distrust `emSign ECC Root CA - G3` until your distribution provides an updated `ca-certificates` package. On older distributions, also distrust `AAA Certificate Services`, which Debian and Ubuntu ship as `Comodo_AAA_Services_root`.

#### Fixes{% #fixes-2 %}

- Fixed an issue where search queries silently dropped operands in `OR` groups that combined exact values with other operands, such as wildcards, ranges, or other attributes. Previously, queries such as `@url:(nope OR aa* OR dummy/api/v1/check_run)` could miss matching events because of the wildcard.
- The Worker now treats regex metacharacters (`.`, `+`, `^`, `$`, `(`, `[`, `|`) in search query wildcard patterns as literal characters. For example, `@url:dummy\/api.v1*` no longer matches `dummy/api/v1/check_run`.

## Worker version 2.24.0{% #worker-version-2240 %}

To upgrade to Worker version 2.24.0:

- Docker: Run the `docker pull` command for the [2.24.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.24.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.24.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.24.0`.

Worker version 2.24.0 gives you access to the following:

#### Security{% #security-3 %}

- The Datadog BYOC Logs destination no longer forwards the Worker's Datadog API key to BYOC Logs.

#### Enhancements{% #enhancements-1 %}

- The Enrichment Table processor:
  - Reports the `enrichment_distinct_lookup_keys_total` metric, which is an estimate of distinct lookup keys summed across 1-second windows.
  - Has improved performance during large cache fills.
  - Reloads table values every 30 seconds. If a reload fails, the processor keeps using the last valid table until a reload succeeds.
  - Caches up to 1 million Reference Table rows locally for 1 hour, which improves handling of large tables.
  - Adds an `outcome` tag to its fetched-keys metric to identify failed lookup attempts.
  - Has a default buffer size of 20,000 events or 100 MiB, and a maximum of 100,000 events.
- The Generate Metrics processor's metric aggregation option is now enabled by default, so generated metrics are aggregated unless you disable the option.
- The Worker now uses `aws-lc-rs` as the cryptography provider for rustls-based TLS connections. The negotiated TLS cipher suites for some connections, such as Databricks Zerobus, might differ from previous versions. OpenSSL-based TLS connections are not affected.
- The Custom Processor has improved performance, including more efficient field lookups and more efficient string functions.
- The Worker has improved configuration startup performance.
- The `pipelines.quota_fill` metric now has a `quota_name` tag. The Worker keeps the legacy `name` tag for backward compatibility.

#### Fixes{% #fixes-3 %}

- The Sensitive Data Scanner processor's library rules now use pattern capture groups to preserve non-sensitive context around the captured value. Previously, the Worker ignored capture groups. For example, the IPv4 address rule now redacts `172.16.0.1:80` as `<REDACTION TEXT>:80` instead of `<REDACTION TEXT>80`.
- The Grok Parser, Parse JSON, and Parse XML processors now log a warning instead of an error when a field fails to parse. The Parse JSON processor's error message also names the field that failed to parse, rather than the generic word `source`.
- Fixed an issue where reloading specific components after configuration updates could cause them to restart unnecessarily during subsequent configuration updates, resetting their state.
- The OpenTelemetry source now preserves native OTLP log event names.
- The Prometheus source now identifies metrics ending with `_total` as counters.
- Fixed an issue with the `keyvalue` filter in the Custom Processor's `parse_grok` and `parse_groks` functions. The filter no longer panics when processing unquoted `NaN` values, such as `ratio=NaN`.

## Worker version 2.23.2{% #worker-version-2232 %}

To upgrade to Worker version 2.23.2:

- Docker: Run the `docker pull` command for the [2.23.2 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.23.2).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.23.2`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.23.2`.

Worker version 2.23.2 gives you access to the following:

#### Enhancements{% #enhancements-2 %}

- The Datadog Logs destination now truncates log events larger than 900 KB. Previously, log events larger than the batch size limit were dropped instead of truncated.

#### Fixes{% #fixes-4 %}

- Fixed the Docker images so the secret generic connector (`sgc`) binary is executable by the `observability-pipelines-worker` service user.

## Worker version 2.23.1{% #worker-version-2231 %}

To upgrade to Worker version 2.23.1:

- Docker: Run the `docker pull` command for the [2.23.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.23.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.23.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.23.1`.

Worker version 2.23.1 gives you access to the following:

#### Security{% #security-4 %}

- Updated the bundled secret generic connector (`sgc`) binary to the version shipped with Datadog Agent 7.84.1, which addresses CVE-2026-56854.

## Worker version 2.23.0{% #worker-version-2230 %}

To upgrade to Worker version 2.23.0:

- Docker: Run the `docker pull` command for the [2.23.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.23.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.23.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.23.0`.

{% alert level="warning" %}
**Breaking change:** This change affects only Custom Processor scripts that use the `contains_all`, `encode_key_value`, `encode_logfmt`, `ip_cidr_contains`, `join`, `parse_groks`, or `tally` functions.  These functions now check the element types of their array arguments when the script compiles. Passing an array with the wrong element type, such as `join([1, 2, 3])`, is now a compile error instead of a runtime failure. When the Worker can confirm the elements are the correct type, the `!` suffix is no longer needed and triggers a warning. For example, use `join(["sources", "transforms", "sinks"], separator: ", ")` instead of `join!(["sources", "transforms", "sinks"], separator: ", ")`.
{% /alert %}

Worker version 2.23.0 gives you access to the following:

#### New features{% #new-features %}

- The Enrichment Table processor now supports CSV files and Reference Tables in metrics pipelines.
- The Custom Processor now supports the `break` statement to exit a `for_each` loop early.

#### Enhancements{% #enhancements-3 %}

- The OpenTelemetry source now limits concurrent requests to 100 and request processing time to 30 seconds by default. The limits apply to both the HTTP and gRPC endpoints. Requests that cannot be handled in time receive a retryable response. Use the `DD_OP_SOURCE_OTEL_MAX_CONCURRENT_REQUESTS` and `DD_OP_SOURCE_OTEL_REQUEST_TIMEOUT_SECS` environment variables to override these limits.
- The Databricks Zerobus destination now identifies the Worker in its `User-Agent` header.
- For the Custom Processor:
  - The processor's overall performance has been improved.
  - The `for_each` function now iterates over collections directly, which improves performance and reduces memory usage.
  - The `md5`, `sha1`, `sha2`, and `sha3` functions now run faster.

#### Fixes{% #fixes-5 %}

- Configuration file paths, such as TLS certificates, enrichment tables, and GCP credentials files, are now resolved against the `data_dir` set in `bootstrap.yaml`. Previously, if `data_dir` was set only in `bootstrap.yaml` and not with the `DD_OP_DATA_DIR` environment variable, the Worker looked for these files in the default data directory and rejected valid files as missing.
- Sources now include the remote `peer_addr` in TCP acknowledgment error logs and TLS connection error logs.
- The Custom Processor's `uuid_v7` function now preserves the supplied timestamp at millisecond precision.

## Worker version 2.22.0{% #worker-version-2220 %}

To upgrade to Worker version 2.22.0:

- Docker: Run the `docker pull` command for the [2.22.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.22.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.22.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.22.0`.

{% alert level="warning" %}
**Breaking change:** This change affects only logs that contain a top-level `attributes` object and are sent to the Datadog Archives destinations (Amazon S3, Google Cloud Storage, and Azure Blob Storage). The change is particularly relevant to logs from the OpenTelemetry source, which commonly include a top-level `attributes` object.  The Datadog Archives destinations store non-[reserved](https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention.md#reserved-attributes) log attributes within an `attributes` object. Starting with Worker 2.22.0, if a log already has an `attributes` object, the Datadog Archives destinations reuse that object instead of nesting it one level deeper. If you use queries with the syntax `@attributes.<field_name>` on logs rehydrated from an Observability Pipelines archive, use `@<field_name>` instead.
{% /alert %}

Worker version 2.22.0 gives you access to the following:

#### New features{% #new-features-1 %}

- The Aggregate processor now supports event-time aggregation, which uses metric timestamps instead of processing time. A configurable grace period includes late-arriving metrics in their event-time window.

#### Enhancements{% #enhancements-4 %}

- The Generate Metrics processor for logs pipelines now has an option to enable metric aggregation. When the option is enabled, generated metrics are rolled up by event timestamp into 1-second buckets. This prevents the Datadog metrics intake from dropping metrics generated within the same second because of last-point-wins behavior.
- Configuration parse errors for a source, processor, or destination now include the component's ID and type, and the field causing the error when it can be determined.
- Reference Table lookup errors caused by a failed HTTP request, such as a timeout, DNS failure, or connection reset, now report the underlying cause instead of a generic `error sending request` message.
- The Datadog BYOC Logs destination now supports a configurable HTTP payload size limit with the `DD_OP_DESTINATION_CLOUDPREM_MAX_PAYLOAD_BYTES` environment variable.
- The following sources now have TCP keepalive enabled by default with a 60-second idle timeout, which helps to detect dead connections when clients disconnect without sending FIN or RST packets:
  - Datadog Agent
  - Fluentd and Fluent Bit
  - Logstash
  - OpenTelemetry (HTTP)
  - Socket (TCP mode)
  - Splunk HEC
  - Syslog (TCP mode)
- The Enrichment Table processor (CSV or file) no longer copies the entire table for every batch of events, which reduces memory and CPU usage for large enrichment tables.

#### Fixes{% #fixes-6 %}

- The Syslog destination no longer shuts down the Worker when a log's `timestamp` field is not a timestamp value, such as a JSON string. These logs now use the current UTC time in the syslog header.
- The `chunked_gelf` framing decoder now limits the payload buffered across incomplete messages to 128 MiB. Previously, an unauthenticated sender could exhaust memory by sending chunks for messages it never completed, especially with the Socket source in UDP mode.
- The Logstash source now rejects frames larger than the configured maximum instead of buffering them indefinitely. Previously, a sender could declare an extremely large frame and force the source to hold its bytes in memory until it ran out of memory.
- The Splunk TCP source now truncates frames that exceed the `DD_OP_SPLUNK_TCP_MAX_FRAME_LENGTH` limit instead of dropping the events. This matches Splunk's `TRUNCATE` behavior.
- Fixed an issue in the `chunked_gelf` framing decoder that caused the source to shut down when a one-byte message arrived and trace-level logging was enabled for the decoder. The decoder now rejects these messages, as it does other malformed payloads.
- Fixed an issue where the internal buffer usage reporter outlived the buffer it reported on, which caused the reporter for the old buffer to keep publishing stale metrics under the same `buffer_id` as its replacement. Metrics for a buffer that was removed, rather than replaced, also now age out under `expire_metrics_secs`, which the continuously republished values previously prevented.
- Fixed the following issues with the `chunked_gelf` framing decoder's limits:
  - `pending_messages_limit` was applied to every chunk instead of only to new messages. After the limit was reached, chunks of messages that were already pending were rejected, so those messages could never complete.
  - Dropping a message for exceeding `max_length` left its timeout task running, so `pending_messages_limit` capped the number of pending messages but not the number of live timeout tasks.
- The FIPS endpoint setting now applies to STS AssumeRole clients. When `AWS_USE_FIPS_ENDPOINT=true` is set, the Worker uses FIPS endpoints, such as `sts-fips.<region>.amazonaws.com`, for STS operations as well as for the primary service client.
- When a compressed request payload fails to decompress, the OpenTelemetry source's gRPC mode now includes the underlying error instead of the generic `reached impossible error during decompressor finalization` message.

#### Removed metrics{% #removed-metrics %}

- The deprecated `pipelines.buffer_byte_size` and `pipelines.buffer_events` gauge metrics have been removed. Use `pipelines.buffer_size_bytes` and `pipelines.buffer_size_events` instead.

## Worker version 2.21.0{% #worker-version-2210 %}

To upgrade to Worker version 2.21.0:

- Docker: Run the `docker pull` command for the [2.21.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.21.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.21.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.21.0`.

Worker version 2.21.0 gives you access to the following:

#### New features{% #new-features-2 %}

- For metrics pipelines:
  - The [Prometheus source](https://docs.datadoghq.com/observability_pipelines/sources/prometheus.md) is now available.
  - The [Prometheus destination](https://docs.datadoghq.com/observability_pipelines/destinations/prometheus.md) is now available.
  - The [OpenTelemetry destination](https://docs.datadoghq.com/observability_pipelines/destinations/opentelemetry/metrics.md) is now available.
- The Datadog Archives destinations (Amazon S3, Google Cloud Storage, and Azure Blob Storage) now support a configurable `compression` option (`gzip` or `zstd`).
- The Remap to OCSF processor now includes:
  - The AWS Config Rules OCSF mapping.
  - The `concat_strings` OCSF mapping function, which joins the string representations of multiple source fields or constant values.
- The Splunk HEC destination now supports an `endpoint_target` option (`event` or `raw`) to determine which Splunk HEC endpoint events are sent to.

#### Enhancements{% #enhancements-5 %}

- The Splunk TCP source now supports configuring TCP keepalive and the connection limit.
- Internal Worker logs and traces are now enriched with the pipeline name (`op_worker.pipeline_name:<pipeline_name>`), matching the tag already present on metrics.
- The Worker's logging during graceful shutdown has been improved.
- The validation of destination endpoints has been improved to check for incomplete or empty URLs.
- The `parse_aws_vpc_flow_log` function in the Custom Processor now recognizes all fields introduced in Amazon VPC Flow Logs versions 7 through 11, including:
  - Version 7: ECS metadata fields
  - Version 8: `reject_reason`
  - Version 9: `resource_id`
  - Version 10: `encryption_status`
  - Version 11: Tag, interface, and next-hop fields
- The Enrichment Table processor now:
  - Keeps rows in its cache for approximately 30 minutes, regardless of whether the lookup was positive or negative
  - Splits the events it yields into batches based on the source sender's batch size

#### Fixes{% #fixes-7 %}

- Fixed case-insensitive glob matching in search queries containing UTF-8 text, which caused panics and incorrect results for non-ASCII input.
- Fixed Azure Blob Storage uploads larger than 4 MiB when using an account-key connection string.
- Fixed an unrecoverable Worker error that occurred when a record exceeded the buffer's maximum record size. The error is now logged and reflected in the `buffer_discarded_events_total` and `buffer_discarded_bytes_total` health metrics (with `intentional="false"`).

## Worker version 2.20.4{% #worker-version-2204 %}

To upgrade to Worker version 2.20.4:

- Docker: Run the `docker pull` command for the [2.20.4 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.20.4).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.20.4`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.20.4`.

Worker version 2.20.4 gives you access to the following:

#### Fixes{% #fixes-8 %}

- The Splunk TCP source previously enforced a hardcoded maximum frame length of 10,000 bytes. The Worker rejected larger events and logged warnings. The source no longer limits the frame length by default. To prevent unbounded memory consumption, use the `DD_OP_SPLUNK_TCP_MAX_FRAME_LENGTH` environment variable to set a maximum frame length.

## Worker version 2.20.3{% #worker-version-2203 %}

To upgrade to Worker version 2.20.3:

- Docker: Run the `docker pull` command for the [2.20.3 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.20.3).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.20.3`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.20.3`.

Worker version 2.20.3 gives you access to the following:

#### Fixes{% #fixes-9 %}

- Fixed the OpenTelemetry source's gRPC listener to advertise `h2` over ALPN when TLS is enabled. Previously, gRPC clients connecting over TLS could fail to establish a connection.

## Worker version 2.20.2{% #worker-version-2202 %}

To upgrade to Worker version 2.20.2:

- Docker: Run the `docker pull` command for the [2.20.2 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.20.2).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.20.2`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.20.2`.

Worker version 2.20.2 gives you access to the following:

#### Fixes{% #fixes-10 %}

- Fixed an issue where non-finite (NaN or Infinity) gauge, counter delta, histogram delta-sum values, and zero-count histograms could be serialized into malformed OTLP metrics payloads. These values are now dropped before OTLP encoding.

## Worker version 2.20.0{% #worker-version-2200 %}

To upgrade to Worker version 2.20.0:

- Docker: Run the `docker pull` command for the [2.20.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.20.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.20.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.20.0`.

Worker version 2.20.0 gives you access to the following:

#### New features{% #new-features-3 %}

- The OpenTelemetry and Splunk HEC sources now support hot-reloading TLS certificates, so certificate rotations are applied without restarting the Worker.

#### Enhancements{% #enhancements-6 %}

- The Split Array processor's performance has been improved.
- The HTTP/S Server source now uses a case-insensitive match for the header name field when you configure authentication tokens.
- The Splunk HEC source now accepts an optional list of tokens for authenticating events.

#### Fixes{% #fixes-11 %}

- The Add Fields processor now increments the `component_errors_total` metric when it fails to update an event.
- Fixed TLS certificate hostname verification issue for destinations with a configured server name, so that the server name is used for verification.

## Worker version 2.19.0{% #worker-version-2190 %}

To upgrade to Worker version 2.19.0:

- Docker: Run the `docker pull` command for the [2.19.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.19.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.19.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.19.0`.

Worker version 2.19.0 gives you access to the following:

#### Enhancements{% #enhancements-7 %}

- A `server_name` TLS option has been added to the BYOC Logs, HTTP Client, Socket, and Syslog destinations, and to the HTTP Client source. It overrides the SNI and certificate hostname used for the TLS handshake, which is applicable when the dialed address does not match the certificate's Common Name or Subject Alternative Name.
- The Splunk TCP source now supports hot-reloading TLS certificates, so certificate rotations are applied without restarting the Worker.
- The BYOC Logs destination now supports hot-reloading mTLS files.
- The Worker now supports a configurable graceful shutdown limit.
- The Amazon S3 destination now supports SSE-KMS encryption using the `server_side_encryption` and `ssekms_key_id` options.

#### Fixes{% #fixes-12 %}

- Fixed an issue in the Reduce processor where a timestamp field whose name requires quoting in a path (for example, `"created.at"` or `"event-time"`) had its `_end` companion field silently dropped from the reduced event. The companion field is now placed correctly next to the base field.
- Fixed a configuration reload issue. If a reload changed a component's type while keeping the same name, such as replacing a source named `X` with a processor named `X`, any downstream processor or destination still reading from `X` now reconnects to the new component.
- The Syslog source (TCP mode) has been fixed to handle an over-length, length-prefixed message split across multiple reads.
- The Logstash source has been fixed to only send an acknowledgment when all events of a batch are received instead of sending a partial acknowledgment.
- The Logstash source now rejects a batch of events that arrives before all events in the current batch has been received, closing the connection with a fatal decode error instead of continuing.
- Fixed the Syslog codec that was silently ignoring short-form severity keywords (`crit`, `emerg`, `err`, `info`, `warn`) and defaulting to `informational`. Both short-form and full-form severity names are now accepted.
- Fixed an issue in the Custom Processor functions `parse_key_value`, `parse_cef`, `decode_mime_q`, and `parse_ruby_hash` that had inputs with lines of 65,535 bytes or more.

## Worker version 2.18.0{% #worker-version-2180 %}

To upgrade to Worker version 2.18.0:

- Docker: Run the `docker pull` command for the [2.18.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.18.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.18.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.18.0`.

Worker version 2.18.0 gives you access to the following:

#### New features{% #new-features-4 %}

- The `run` command now supports a `--log-format` flag, also configurable using the `DD_OP_LOG_FORMAT` environment variable, for selecting the format of the Worker's own `stdout` and `stderr` logs. The default `text` preserves the existing human-readable output, and `json` emits structured JSON logs that can be tailed and parsed directly without additional parsing rules.
- The Tag Cardinality Limit processor now supports a `tracking_mode` option. `exact_fingerprint` tracks tag values exactly using 64-bit fingerprints, which uses less memory than the previous method of storing raw values, and `probabilistic` uses a bloom filter for even lower memory usage in exchange for an occasional false positive. Probabilistic mode accepts a `false_positive_rate` field, which defaults to `0.001` (0.1%).
- The WebSocket source is available for ingesting logs from a WebSocket endpoint, with support for `none`, `basic`, `bearer`, and `custom` authentication strategies, as well as TLS.
- A new `generate_metrics` processor routes metrics to any supported metrics destination. Datadog recommends using it over the existing `generate_datadog_metrics` processor, although use of the latter is not affected.

#### Enhancements{% #enhancements-8 %}

- mTLS support has been added to the BYOC Logs destination.
- The `measure_cpu_usage` option is now enabled, so that `pipelines.component_cpu_usage_ns_total` is always emitted for the following log processors: Custom Processor, Sensitive Data Scanner, Grok Parser, Parse JSON, Parse XML, OCSF Mapper, Enrichment Table, Reduce, Dedupe, Split Array, and Throttle; and the following metrics processors: Aggregate and Tag Cardinality Limit.
- Pipeline configuration errors now include the ID of the component that caused the failure. This makes it easier to identify which source, processor, or destination needs correcting. For processors made up of multiple blocks, such as the Custom Processor, Generate Metrics, and OCSF Mapper, the error also identifies which individual block failed, including its position and, where the block has one, its name.
- Pipeline metrics are now tagged with the pipeline name using `pipeline_name:<NORMALIZED_PIPELINE_NAME>`.

#### Fixes{% #fixes-13 %}

- Fixed an issue that occurred when a log-only pipeline (using legacy search syntax) ran a search query that the new parser could not handle. The parser now emits a warning and does not run the invalid query.
- Fixed an issue affecting queries that start with an escaped character (for pipelines running new search syntax only).
- Internal telemetry (metrics and logs) emitted from background tasks now correctly inherits the owning component's tags (`component_id`, `component_kind`, `component_type`). Previously, several components spawned background tasks without propagating their tags, so some internal events from those tasks were missing their component tags. Affected emissions include:
  - Datadog archive source: listing-error, download-error, send-error, event-acknowledged, event-errored, and event-rejected, and status metrics from its listing coordinator, per-file downloader, and acknowledgment-handler tasks
  - Microsoft Sentinel destination: Azure OAuth token-regeneration task
  - Splunk TCP source: server task.

## Worker version 2.17.0{% #worker-version-2170 %}

To upgrade to Worker version 2.17.0:

- Docker: Run the `docker pull` command for the [2.17.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.17.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.17.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.17.0`.

{% alert level="warning" %}
**Breaking change:** The Generate Metrics processor now emits the raw field name as the tag key for `group_by` entries that contain characters outside of `[a-zA-Z0-9_@]`, such as `http.status_code` and `kube-cronjob`. Previously, the processor added a trailing underscore (`_`) to tag keys that contained those characters. For example, the tag key `http.status_code` became `http.status_code_` because the key contained a period (`.`).  After upgrading to Worker 2.17, update any dashboards, monitors, alerts, or notebooks that use tag keys with the trailing underscore to use those tag keys without the underscore. Historical datapoints remain queryable under the old tag key. Pipelines whose `group_by` entries only use names matching `[a-zA-Z0-9_@]+` are not affected.
{% /alert %}

Worker version 2.17.0 gives you access to the following:

#### New features{% #new-features-5 %}

- For the Custom Processor:
  - VRL string literals now support `\u{HEX}` Unicode escape sequences. Any valid Unicode scalar value can be expressed, such as `"hello\u{1F30E}world"`. Invalid sequences, such as empty braces, non-hex digits, surrogate codepoints, or values above U+10FFFF, are reported as compile-time errors.
  - The `parse_regex` function now accepts dynamic regex patterns (variables and runtime expressions), consistent with `parse_regex_all`. When the pattern is a literal, return type information remains precise based on named capture groups.

#### Enhancements{% #enhancements-9 %}

- The Generate Metrics processor's performance has been improved to reduce CPU overhead when using `group_by` tag labels.
- For the Custom Processor, the performance of `parse_regex_all` has been improved by reusing the compiled regex across invocations.

#### Fixes{% #fixes-14 %}

- HTTP metrics emitted by the Quota processor's background sync now have correct component tags (`component_kind:transform`, `component_type:quota`, `component_id:quota_global_state`).
- The Enrichment Table processor using Reference Tables now skips sending empty event batches, preventing fatal errors with disk buffers.
- The Generate Metrics processor now uses a static component ID so that associated metrics share the same `component_id` across Workers and restarts.
- An issue with parsing filter queries that contain whitespace inside parentheses, such as `service:( web OR api )`, has been fixed.
- Live Capture now works correctly for sources with multiple named output ports, such as the OpenTelemetry source.
- An issue where a Worker crash could occur if a source or a processor sends an empty event batch to the next component has been fixed.
- For the Custom Processor:
  - Error messages and unused variable diagnostics have been fixed. The processor now reports every unhandled error in a single compilation.
  - You can now use `SCREAMING_SNAKE` case in functions such as `pascalcase` and `camelcase`.
  - The `encode_proto` and `parse_proto` functions now support proto maps whose keys are integers or Booleans, not only strings.

## Worker version 2.16.1{% #worker-version-2161 %}

To upgrade to Worker version 2.16.1:

- Docker: Run the `docker pull` command for the [2.16.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.16.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.16.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.16.1`.

Worker version 2.16.1 gives you access to the following:

#### Fixes{% #fixes-15 %}

- The Worker no longer fails when fetching configurations in non-US1 data centers.

## Worker version 2.16.0{% #worker-version-2160 %}

To upgrade to Worker version 2.16.0:

- Docker: Run the `docker pull` command for the [2.16.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.16.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.16.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.16.0`.

Worker version 2.16.0 gives you access to the following:

#### New features{% #new-features-6 %}

- [Databricks Zerobus](https://docs.datadoghq.com/observability_pipelines/destinations/databricks.md) destination: Send log data to Databricks Unity Catalog tables using the Zerobus ingestion service. The Databricks destination supports OAuth 2.0 authentication, automatic schema fetching from Unity Catalog, and protobuf batch encoding.
- The Splunk HEC source now supports enriching incoming log events using the VRL decoder.
- The Amazon S3 destination now supports Apache Parquet batch encoding with flexible schema definitions and configurable compression (Snappy, ZSTD, GZIP, LZ4, or none).
- For the Custom Processor:
  - The `encode_proto` function now accepts an `allow_lossy_string_coercion` argument.
  - Protobuf encoding now coerces compatible scalar types into the target field type: integers and strings are accepted for `bool` fields, and integers are accepted for `float` or `double` fields.

#### Enhancements{% #enhancements-10 %}

- mTLS support has been added to the following sources: Fluent, Logstash, OpenTelemetry (logs and metrics), Splunk HEC, Splunk TCP, HTTP/S Server, Socket, and Syslog.
- Live Capture events now include the UUID of the Worker that sent the event.
- The Reference Tables processor's buffer now emits `buffer_size_events` and `buffer_size_bytes` gauge metrics to replace the deprecated `buffer_events` and `buffer_byte_size` metrics. The deprecated metrics are still being sent for backward compatibility.
- The Datadog Logs destination now supports overrides for per-site logs endpoint using the bootstrap `logs-sites` field or `DD_OP_LOGS_<SITE>` environment variable, such as `DD_OP_LOGS_US1`. Environment variables take precedence over the bootstrap file.
- Sources now record these distribution metrics:
  - `source_send_latency_seconds`: The time it takes for the source to send a chunk of events to the next component.
  - `source_send_batch_latency_seconds`: The time it takes for the source to send a batch, which can contain multiple event chunks, to the next component.

#### Fixes{% #fixes-16 %}

- A race condition in the Reference Tables processor has been fixed to prevent dropping buffered events during a Worker shutdown.
- Live Capture was updated to improve handling of bursty traffic and metrics from the Generate Metrics processor.
- The Worker no longer logs `Root metadata expired` or `potential freeze attack` on startup after refreshing embedded Remote Config trusted-root metadata.
- The Splunk HEC source now emits `authentication_failed` as the `error_type` in error logs and metrics when authentication fails due to a missing or invalid authorization header.
- Fixed the Datadog Logs destination health check endpoint computation to preserve site prefixes, such as `us3.`, `us5.`, `ap1.`, when deriving the API URL from intake endpoints.
- An issue where a destination with a configured disk buffer would stall for `batch.timeout_sec` before gracefully reloading has been fixed. This fix also resolves cases where the Worker ignored SIGINT during a pipeline stall.
- Fixed the Custom Processor so an `else` or `else if` keyword can be on a new line after the closing curly brace (`}`) of an `if`-block.

## Worker version 2.15.1{% #worker-version-2151 %}

To upgrade to Worker version 2.15.1:

- Docker: Run the `docker pull` command for the [2.15.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.15.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.15.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.15.1`.

Worker version 2.15.1 gives you access to the following:

#### New features{% #new-features-7 %}

- The Custom Processor now supports the `to_entries` and `from_entries` functions for converting between objects and arrays of key-value pairs (jq-style).

#### Enhancements{% #enhancements-11 %}

- The Datadog Metrics destination now uses zstd compression for Datadog API v2 and sketches endpoints, reducing CPU, memory, and bandwidth usage.
- The `flatten` function in the Custom Processor now accepts an optional `except` argument to exclude specific keys from flattening.

#### Fixes{% #fixes-17 %}

- A performance regression in buffer metrics tracking has been fixed.
- The Datadog Agent source has been fixed to preserve the `device` tag from Datadog API v2 resources.

## Worker version 2.15.0{% #worker-version-2150 %}

To upgrade to Worker version 2.15.0:

- Docker: Run the `docker pull` command for the [2.15.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.15.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.15.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.15.0`.

Worker version 2.15.0 gives you access to the following:

#### New features{% #new-features-8 %}

- New OCSF mappings have been added for the following log types:
  - AWS GuardDuty for all finding types (for example, EKS Audit, EC2, Lambda, IAM, DNS) in a single mapping named `AWS GuardDuty`.
  - Infoblox NIOS: DNS Activity, DHCP Activity, audit (Authentication and API Activity), and port and syslog Base Event (Infoblox DNS Query, Infoblox DHCP, Infoblox Audit Authentication, Infoblox Audit API, Infoblox Port).
  - Zscaler ZPA App Connector Status logs to OCSF schema version 1.3.0 (Authentication, class 3002) with `datetime` and `host` profiles.
  - Zscaler ZPA User Activity logs to OCSF schema version 1.3.0 (Network Activity, class 4001) with `datetime`, `host`, `network_proxy`, and `security_control` profiles.
  - AWS WAF Web ACL logs. Transforms WAF log events into OCSF HTTP Activity (class 4002) with `cloud` and `security_control` profiles.
  - Zscaler ZPA User Status logs to OCSF schema version 1.3.0 (Authentication, class 3002) with `datetime` and `host` profiles.
- The OpenTelemetry source now supports metrics pipelines.
- The Elasticsearch destination is now available for metrics pipelines.
- The `parse_yaml` function is now available for the Custom Processor. This function parses YAML according to the [YAML 1.1 spec](https://yaml.org/spec/1.1/).
- The Enrichment Table file option now supports a `field` option that accepts an event field path or a metadata secret as the lookup key source. Plain text is supported for backwards compatibility.
  - Explicit event path example:
    - Observability Pipelines simplified syntax: `field: {event: "message"}`
    - VRL syntax for the Custom Processor: `field: {vrl: ".message"}`
  - Metadata secret reference example: `field: {secret: "splunk_hec_token"}`
  - Plain string (for backwards compatibility):
    - Observability Pipelines simplified syntax: `field: "message"`
    - VRL syntax for the Custom Processor: `field: ".message"`

#### Enhancements{% #enhancements-12 %}

- The Amazon S3 source now accepts compressed data.
- The Elasticsearch destination has been updated with new options: `auto_routing`, `compression`, `id_key`, `pipeline`, `request_retry_partial`, `sync_fields`, and `tls`.
- Mapping array-of-object source fields into OCSF array-of-object destinations is now supported.
- The Datadog Metrics destination now defaults to the Datadog series v2 endpoint (`/api/v2/series`).
- The Enrichment Table processor's GeoIP option now includes a network field containing the CIDR network associated with the lookup result. The network field is available for all database types (City, ISP/ASN, Connection-Type, Anonymous-IP).
- The Custom Processor now has an `encode_csv` function that encodes an array of values into a CSV-formatted string. This is the inverse of the `parse_csv` function and supports an optional single-byte delimiter (defaults to `,`).
- Field names now support `.`, such as `foo."bar.baz"`.

#### Fixes{% #fixes-18 %}

- Improved the accuracy of the buffer utilization metric tracking.
- For the Custom Processor, incorrect parameter types for `floor`, `md5`, `parse_key_value`, `precision`, and `seahash` have been fixed.

## Worker version 2.14.1{% #worker-version-2141 %}

To upgrade to Worker version 2.14.1:

- Docker: Run the `docker pull` command for the [2.14.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.14.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.14.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.14.1`.

Worker version 2.14.1 gives you access to the following:

#### Fixes{% #fixes-19 %}

- Fixed how an empty path in a processor field is handled. For example, how the Parse JSON processor handles the `Field to parse JSON on` with the value `.`.

## Worker version 2.14.0{% #worker-version-2140 %}

To upgrade to Worker version 2.14.0:

- Docker: Run the `docker pull` command for the [2.14.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.14.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.14.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.14.0`.

Worker version 2.14.0 gives you access to the following:

#### New features{% #new-features-9 %}

- OCSF mappings for Palo Alto Networks Threat events have been added.
- The Database source has been updated with timeout-related changes.
- The `component_latency_seconds` histogram and `component_latency_mean_seconds` gauge internal metrics have been added. The metrics are based on the time an event spends in a single processor, including in the processor buffer.

#### Enhancements{% #enhancements-13 %}

- Enrichment Table error reporting now uses Reference Tables metrics to reduce the count of similar logs.
- The Splunk HEC destination now supports extracting index fields from events.
- The OCSF mapper now has an option to retain unmatched fields.
- For the Enrichment Table processor, the local cache retention time of entries not found in a Reference Table has been increased. The retention time is now 30 minutes, up from 10 minutes.
- The Database Source SQL validation checks have been improved.
- The Sensitive Data Scanner library now has new and updated out-of-the-box scanning rules for PII, credentials, and financial data. Minor bugs have also been fixed.
- The `observability-pipelines-worker top` command has new keybinds for scrolling, sorting, and filtering.
- The Datadog Logs destination has been updated to default to `zstd` compression instead of no compression.
- The environment variable for the Datadog Agent source address is now configurable.

#### Fixes{% #fixes-20 %}

- Fixed a bug with sticky error state when Remote Configuration is successfully polled.
- Fixed buffer utilization metrics to properly record actual utilization level.
- Fixed a Worker shutdown race condition between closing the memory buffer and in-progress send operations that could potentially cause event loss.
- The Generate Metrics processor now handles aggregated histogram and aggregated summary metrics correctly.
- Live Capture now supports child events in the split array processor.
- Reference Tables buffer size and request frequency have been reduced to avoid out-of-memory (OOM) and rate limit errors.
- The Reference Tables processor now rejects empty or blank lookup keys and supports integer keys.

## Worker version 2.13.2{% #worker-version-2132 %}

To upgrade to Worker version 2.13.2:

- Docker: Run the `docker pull` command for the [2.13.2 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.13.2).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.13.2`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.13.2`.

Worker version 2.13.2 gives you access to the following:

#### Fixes{% #fixes-21 %}

- Fixed `exists` and `missing` queries to match with objects.

## Worker version 2.13.1{% #worker-version-2131 %}

To upgrade to Worker version 2.13.1:

- Docker: Run the `docker pull` command for the [2.13.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.13.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.13.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.13.1`.

Worker version 2.13.1 gives you access to the following:

#### Fixes{% #fixes-22 %}

- All processors have been updated to gracefully handle incorrect filter query syntax.

## Worker version 2.13.0{% #worker-version-2130 %}

To upgrade to Worker version 2.13.0:

- Docker: Run the `docker pull` command for the [2.13.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.13.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.13.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.13.0`.

Worker version 2.13.0 gives you access to the following:

#### New features{% #new-features-10 %}

- [Custom Processor](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor.md) for metrics: Use VRL to transform metric events.
- [Secrets Management](https://docs.datadoghq.com/observability_pipelines/configuration/secrets_management.md): Observability Pipelines can retrieve secrets using Datadog Secrets Management.
- [Live capture](https://docs.datadoghq.com/observability_pipelines/configuration/live_capture.md) is available for metrics pipelines.
- The [Enrichment Tables](https://docs.datadoghq.com/observability_pipelines/processors/enrichment_table.md) processor can use datasets in Reference Tables.

#### Enhancements{% #enhancements-14 %}

- [Disk buffers](https://docs.datadoghq.com/observability_pipelines/scaling_and_performance/buffering_and_backpressure.md#destination-buffers) have been updated to drop logs when the buffer is full.
- The Dedupe processor has been updated with a configurable cache size.
- The Datadog Agent source has been updated with configurable request timeouts.
- Source buffers have been updated to record the utilization level of the buffer with these metrics:
  - `source_buffer_max_byte_size`
  - `source_buffer_max_event_size`
  - `source_buffer_utilization`
  - `source_buffer_utilization_level`
- Processor buffers have been updated to record the utilization level of the buffers with these metrics:
  - `transform_buffer_max_byte_size`
  - `transform_buffer_max_event_size`
  - `transform_buffer_utilization`
  - `transform_buffer_utilization_level`
- The TLS implementation has been updated to store credentials in FIPS-compliant PEM format.

#### Fixes{% #fixes-23 %}

- Live Capture has been updated and bugs have been fixed.
- The Search Syntax bug with handling hyphenated segments has been fixed.
- The syslog source in UDP mode emits the standard `component_received` metrics, like how it does with TCP mode:
  - `component_received_events_total`
  - `component_received_event_bytes_total`
  - `component_received_bytes_total`

## Worker version 2.12.0{% #worker-version-2120 %}

To upgrade to Worker version 2.12.0:

- Docker: Run the `docker pull` command for the [2.12.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.12.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.12.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.12.0`.

Worker version 2.12.0 gives you access to the following:

#### New features{% #new-features-11 %}

- [HTTP destination](https://docs.datadoghq.com/observability_pipelines/destinations/http_client.md) for metrics pipelines: Routes metrics to an HTTP client endpoint.
- [MySQL Source](https://docs.datadoghq.com/observability_pipelines/sources/mysql.md): Sends logs from a MySQL database to Observability Pipelines.

#### Enhancements{% #enhancements-15 %}

- The HTTP Client source and destination have been updated so you can set a custom authorization strategy.
- The metrics filter processor was updated to filter metrics on `kind` and `value`.
- Processor groups that route and process only targeted events have been updated to reduce processing overhead.
- The Datadog Agent source has been updated to support timeouts, incrementing the `component_timed_out_events_total` and `component_timed_out_requests_total` metrics.

#### Fixes{% #fixes-24 %}

- The Amazon S3 destination has been updated to ensure the `message` field is always a string, JSON-encoding it if necessary.
- A Worker bug has been fixed to ensure Worker logs are reported correctly.
- The `hostname` is renamed to `host` when sending logs to Datadog Archives.
- For metrics sources, Workers have been updated to use their own copy of the Datadog key for authentication, disregarding any keys sent in by the Datadog Agent to prevent the use of stale keys.
- The Worker uses proxy settings configured with environment variables (for example, the `DD_PROXY_HTTPS` environment variable) or in the bootstrap file when it publishes events to Live Capture.

## Worker Version 2.11.0{% #worker-version-2110 %}

To upgrade to Worker version 2.11.0:

- Docker: Run the `docker pull` command for the [2.11.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.11).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.11.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.11.0`.

{% alert level="info" %}
For pipelines that are running Worker 2.10 or older: - After you upgrade to Worker 2.11, your processor filter queries continue to run the legacy search syntax. - You must manually update your filter queries to the new Search Syntax. - Then, enable the New Search Syntax toggle in the UI, or set `use_legacy_search_syntax` to `false` using the API or Terraform.  See [Upgrade Your Filter Queries to the New Search Syntax](https://docs.datadoghq.com/observability_pipelines/guide/upgrade_your_filter_queries_to_the_new_search_syntax.md) for more information.
{% /alert %}

Version 2.11.0 gives you access to the following:

#### New features{% #new-features-12 %}

- More than 100 out-of-the-box rules for the Sensitive Data Scanner processor have been added. These rules redact Personally Identifiable Information (PII) and access key information.
- The updated [Search Syntax](https://docs.datadoghq.com/observability_pipelines/search_syntax/logs.md) that lets you:
  - Dereference arrays
  - Perform case insensitive search within log messages
  - Deterministically target log attributes without using `@` symbol

## Worker Version 2.10.0{% #worker-version-2100 %}

To upgrade to Worker version 2.10.0:

- Docker: Run the `docker pull` command for the [2.10.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.10).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.10.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.10.0`.

Worker version 2.10.0 gives you access to the following:

#### New features{% #new-features-13 %}

- [Kafka destination](https://docs.datadoghq.com/observability_pipelines/destinations/kafka.md): Send logs from Observability Pipelines to your Kafka topics.
- New and updated [Custom Processor functions](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor.md#custom-functions):
  - The `pop` function removes the last item from an array.
  - The cryptographic functions `encrypt_ip` and `decrypt_ip` for IP address encryption.
    - These functions use the IPCrypt specification and support both IPv4 and IPv6 addresses with two encryption modes:
      - aes128 (IPCrypt deterministic, 16-byte key)
      - pfx (IPCryptPfx, 32-byte key).
      - Both algorithms are format-preserving (output is a valid IP address) and deterministic.
  - The `xxhash` function implements `xxh32`, `xxh64`, `xxh3_64`, and `xxh3_128` hashing algorithms.
  - The `parse_aws_alb_log` function has been updated with an optional `strict_mode` parameter.
    - When `strict_mode` is set to `false`, the parser ignores any newly added or trailing fields in AWS ALB logs, instead of failing.
    - Defaults to `true` to preserve current behavior.
  - [Metrics pipelines](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md?tab=metrics#set-up-a-pipeline-in-the-ui):
    - [Datadog Agent source](https://docs.datadoghq.com/observability_pipelines/sources/datadog_agent.md?tab=metrics): Send metrics from the Datadog Agent to Observability Pipelines for processing.
    - [Filter processor](https://docs.datadoghq.com/observability_pipelines/processors/filter.md?tab=metrics): Filter the metrics you want to process.
    - [Tag processor](https://docs.datadoghq.com/observability_pipelines/processors/tag_allow_block_list.md): Include or exclude specific tags in your metrics.
    - [Datadog Metrics destination](https://docs.datadoghq.com/observability_pipelines/destinations/datadog_metrics.md?tab=secretsmanagement): Send your processed metrics Datadog.

#### Enhancements{% #enhancements-16 %}

- The Custom Processor's performance has been improved.
- Workers have been updated to use their own copy of the Datadog key for authentication, disregarding any keys sent in by the Datadog Agent to prevent the use of stale keys.
- Error reporting has been improved when validating JSON schema in custom functions that use the `validate_json_schema` function.

#### Fixes{% #fixes-25 %}

- Group-level filtering logic has been fixed to exclude correct logs.

## Worker Version 2.9.1{% #worker-version-291 %}

To upgrade to Worker version 2.9.1:

- Docker: Run the `docker pull` command to pull the [2.9.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.9.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.9.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.9.1`.

Worker version 2.9.1 gives you access to the following:

#### Fixes{% #fixes-26 %}

- The Microsoft Sentinel destination has been limited to batch sizes of 1 MB when reading logs using the Azure Logs Ingestion API. The limit size was determined based on the [Azure documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits#logs-ingestion-api).

## Worker Version 2.9.0{% #worker-version-290 %}

To upgrade to Worker version 2.9.0:

- Docker: Run the `docker pull` command to pull the [2.9.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.9.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.9.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.9.0`.

Worker version 2.9.0 gives you access to the following:

#### New features{% #new-features-14 %}

- [OpenTelemetry Collector source](https://docs.datadoghq.com/observability_pipelines/sources/opentelemetry.md): Ingest logs from your OpenTelemetry Collector into Observability Pipelines.
- [Datadog BYOC Logs destination](https://docs.datadoghq.com/observability_pipelines/destinations/datadog_byoc_logs.md): Route logs to the Datadog BYOC Logs destination.
- [Google Pub/Sub destination](https://docs.datadoghq.com/observability_pipelines/destinations/google_pubsub.md): Send logs from Observability Pipelines to the Google Pub/Sub messaging system.
- The `haversine` custom function to calculate haversine distance and bearing.

#### Enhancements{% #enhancements-17 %}

- The Observability Pipelines Worker's internal logs have been updated to partially redact the Datadog API key (first 28 characters only), to help investigate API-key related issues.
- The performance of Remote Configuration delivery time has been improved.
- The `parse_cef` and `parse_syslog` custom functions have enhanced parsing.

## Worker Version 2.8.1{% #worker-version-281 %}

To upgrade to Worker version 2.8.1:

- Docker: Run the `docker pull` command to pull the [2.8.1 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.8.1).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.8.1`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.8.1`.

Worker version 2.8.1 gives you access to the following:

#### Fixes{% #fixes-27 %}

- The HTTP Client source's authorization strategy has been fixed.

## Worker Version 2.8.0{% #worker-version-280 %}

To upgrade to Worker version 2.8.0:

- Docker: Run the `docker pull` command to pull the [2.8.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.8.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.8.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.8.0`.

Worker version 2.8.0 gives you access to the following:

#### New features{% #new-features-15 %}

- All sources and destinations have been updated to support custom environment variables.

#### Enhancements{% #enhancements-18 %}

- The Elasticsearch destination's [indexing strategy](https://docs.datadoghq.com/observability_pipelines/destinations/elasticsearch.md#set-up-the-destination) has been updated to include data streams.
- The HTTP Client destination supports template syntax.

#### Fixes{% #fixes-28 %}

- The HTTP Server source's TLS enablement has been fixed.
- Worker health metrics have been fixed.
- OpenSearch's basic authentication has been fixed.

## Worker Version 2.7.0{% #worker-version-270 %}

To upgrade to Worker version 2.7.0:

- Docker: Run the `docker pull` command to pull the [2.7.0 image](https://hub.docker.com/r/datadog/observability-pipelines-worker/tags?name=2.7.0).
- Kubernetes: See the [Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/observability-pipelines-worker#observability-pipelines-worker) and [Upgrade the Worker](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker.md?interface=ui&platform=kubernetes&secrets_source=secrets_management#upgrade-the-worker-4).
- APT: Run the command `apt-get install observability-pipelines-worker=2.7.0`.
- RPM: Run the command `sudo yum install observability-pipelines-worker-2.7.0`.

Worker version 2.7.0 gives you access to the following:

#### New features{% #new-features-16 %}

- [The HTTP Client destination](https://docs.datadoghq.com/observability_pipelines/destinations/http_client.md): Send logs to an HTTP Client, such as a logging platform or SIEM.
- [Processor Groups](https://docs.datadoghq.com/observability_pipelines/processors.md#processor-groups): Organize your processors into logical groups to help you manage them.
- [Disk and memory](https://docs.datadoghq.com/observability_pipelines/scaling_and_performance/buffering_and_backpressure.md#destination-buffers) buffering options are available for destinations.

#### Enhancements{% #enhancements-19 %}

- The `decode_lz4` custom function has been updated to support decompressing `lz4` frame data.
- The Azure Blob Storage and Google Cloud Storage archive destinations' prefix fields support template syntax.
- The Splunk HEC destination has a custom environment variable.
- The sample processor has an optional [`group_by` parameter](https://docs.datadoghq.com/observability_pipelines/processors/sample.md#group-by-example).

#### Fixes{% #fixes-29 %}

- The Datadog Logs destination's default compression has been updated to `zstd`, which matches Datadog Agent's default compression.
- The Amazon S3, Google Cloud Storage, and Azure Blob Storage destinations have been fixed to resolve log timestamps correctly.
- The custom OCSF mapper's performance has been improved.
- The filter processor has flag logic enabled to pass events to the next processor.