Upgrade the Worker Guide

This product is not supported for your selected Datadog site. ().
このページは日本語には対応しておりません。随時翻訳に取り組んでいます。
翻訳に関してご質問やご意見ございましたら、お気軽にご連絡ください。

Overview

Datadog recommends updating the Observability Pipelines Worker (OPW) with every minor and patch release, or monthly at a minimum.

Upgrading to the latest major OPW version and keeping it updated is the only supported way to get new OPW functionalities, fixes, and security updates.

This guide goes over how to upgrade to a specific Worker version and the updates for that version.

Worker version 2.13.1

To upgrade to Worker version 2.13.1:

  • Docker: Run the docker pull command for the 2.13.1 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.13.1.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.13.1.

Worker version 2.13.1 gives you access to the following:

Fixes

  • All processors have been updated to gracefully handle incorrect filter query syntax.

Worker version 2.13.0

To upgrade to Worker version 2.13.0:

  • Docker: Run the docker pull command for the 2.13.0 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.13.0.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.13.0.

Worker version 2.13.0 gives you access to the following:

New features

Enhancements

  • Disk buffers have been updated to drop logs when the buffer is full.
  • The Dedupe processor has been updated with a configurable cache size.
  • The Datadog Agent source has been updated with configurable request timeouts.
  • Source buffers have been updated to record the utilization level of the buffer with these metrics:
    • source_buffer_max_byte_size
    • source_buffer_max_event_size
    • source_buffer_utilization
    • source_buffer_utilization_level
  • Processor buffers have been updated to record the utilization level of the buffers with these metrics:
    • transform_buffer_max_byte_size
    • transform_buffer_max_event_size
    • transform_buffer_utilization
    • transform_buffer_utilization_level
  • The TLS implementation has been updated to store credentials in FIPS-compliant PEM format.

Fixes

  • Live Capture has been updated and bugs have been fixed.
  • The Search Syntax bug with handling hyphenated segments has been fixed.
  • The syslog source in UDP mode emits the standard component_received metrics, like how it does with TCP mode:
    • component_received_events_total
    • component_received_event_bytes_total
    • component_received_bytes_total

Worker version 2.12.0

To upgrade to Worker version 2.12.0:

  • Docker: Run the docker pull command for the 2.12.0 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.12.0.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.12.0.

Worker version 2.12.0 gives you access to the following:

New features

  • HTTP destination for metrics pipelines: Routes metrics to an HTTP client endpoint.
  • MySQL Source: Sends logs from a MySQL database to Observability Pipelines.

Enhancements

  • The HTTP Client source and destination have been updated so you can set a custom authorization strategy.
  • The metrics filter processor was updated to filter metrics on kind and value.
  • Processor groups that route and process only targeted events have been updated to reduce processing overhead.
  • The Datadog Agent source has been updated to support timeouts, incrementing the component_timed_out_events_total and component_timed_out_requests_total metrics.

Fixes

  • The Amazon S3 destination has been updated to ensure the message field is always a string, JSON-encoding it if necessary.
  • A Worker bug has been fixed to ensure Worker logs are reported correctly.
  • The hostname is renamed to host when sending logs to Datadog Archives.
  • For metrics sources, Workers have been updated to use their own copy of the Datadog key for authentication, disregarding any keys sent in by the Datadog Agent to prevent the use of stale keys.
  • The Worker uses proxy settings configured with environment variables (for example, the DD_PROXY_HTTPS environment variable) or in the bootstrap file when it publishes events to Live Capture.

Worker Version 2.11.0

To upgrade to Worker version 2.11.0:

  • Docker: Run the docker pull command for the 2.11.0 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.11.0.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.11.0.
For pipelines that are running Worker 2.10 or older:
- After you upgrade to Worker 2.11, your processor filter queries continue to run the legacy search syntax.
- You must manually update your filter queries to the new Search Syntax.
- Then, enable the New Search Syntax toggle in the UI, or set use_legacy_search_syntax to false using the API or Terraform.

See Upgrade Your Filter Queries to the New Search Syntax for more information.

Version 2.11.0 gives you access to the following:

New features

  • More than 100 out-of-the-box rules for the Sensitive Data Scanner processor have been added. These rules redact Personally Identifiable Information (PII) and access key information.
  • The updated Search Syntax that lets you:
    • Dereference arrays
    • Perform case insensitive search within log messages
    • Deterministically target log attributes without using @ symbol

Worker Version 2.10.0

To upgrade to Worker version 2.10.0:

  • Docker: Run the docker pull command for the 2.10.0 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.10.0.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.10.0.

Worker version 2.10.0 gives you access to the following:

New features

  • Kafka destination: Send logs from Observability Pipelines to your Kafka topics.
  • New and updated Custom Processor functions:
    • The pop function removes the last item from an array.
    • The cryptographic functions encrypt_ip and decrypt_ip for IP address encryption.
      • These functions use the IPCrypt specification and support both IPv4 and IPv6 addresses with two encryption modes:
        • aes128 (IPCrypt deterministic, 16-byte key)
        • pfx (IPCryptPfx, 32-byte key).
        • Both algorithms are format-preserving (output is a valid IP address) and deterministic.
    • The xxhash function implements xxh32, xxh64, xxh3_64, and xxh3_128 hashing algorithms.
    • The parse_aws_alb_log function has been updated with an optional strict_mode parameter.
      • When strict_mode is set to false, the parser ignores any newly added or trailing fields in AWS ALB logs, instead of failing.
      • Defaults to true to preserve current behavior.
    • Metrics pipelines:

Enhancements

  • The Custom Processor’s performance has been improved.
  • Workers have been updated to use their own copy of the Datadog key for authentication, disregarding any keys sent in by the Datadog Agent to prevent the use of stale keys.
  • Error reporting has been improved when validating JSON schema in custom functions that use the validate_json_schema function.

Fixes

  • Group-level filtering logic has been fixed to exclude correct logs.

Worker Version 2.9.1

To upgrade to Worker version 2.9.1:

  • Docker: Run the docker pull command to pull the 2.9.1 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.9.1.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.9.1.

Worker version 2.9.1 gives you access to the following:

Fixes

  • The Microsoft Sentinel destination has been limited to batch sizes of 1 MB when reading logs using the Azure Logs Ingestion API. The limit size was determined based on the Azure documentation.

Worker Version 2.9.0

To upgrade to Worker version 2.9.0:

  • Docker: Run the docker pull command to pull the 2.9.0 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.9.0.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.9.0.

Worker version 2.9.0 gives you access to the following:

New features

Enhancements

  • The Observability Pipelines Worker’s internal logs have been updated to partially redact the Datadog API key (first 28 characters only), to help investigate API-key related issues.
  • The performance of Remote Configuration delivery time has been improved.
  • The parse_cef and parse_syslog custom functions have enhanced parsing.

Worker Version 2.8.1

To upgrade to Worker version 2.8.1:

  • Docker: Run the docker pull command to pull the 2.8.1 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.8.1.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.8.1.

Worker version 2.8.1 gives you access to the following:

Fixes

  • The HTTP Client source’s authorization strategy has been fixed.

Worker Version 2.8.0

To upgrade to Worker version 2.8.0:

  • Docker: Run the docker pull command to pull the 2.8.0 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.8.0.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.8.0.

Worker version 2.8.0 gives you access to the following:

New features

  • All sources and destinations have been updated to support custom environment variables.

Enhancements

  • The Elasticsearch destination’s indexing strategy has been updated to include data streams.
  • The HTTP Client destination supports template syntax.

Fixes

  • The HTTP Server source’s TLS enablement has been fixed.
  • Worker health metrics have been fixed.
  • OpenSearch’s basic authentication has been fixed.

Worker Version 2.7.0

To upgrade to Worker version 2.7.0:

  • Docker: Run the docker pull command to pull the 2.7.0 image.
  • Kubernetes: See the Helm chart.
  • APT: Run the command apt-get install observability-pipelines-worker=2.7.0.
  • RPM: Run the command sudo yum install observability-pipelines-worker-2.7.0.

Worker version 2.7.0 gives you access to the following:

New features

Enhancements

  • The decode_lz4 custom function has been updated to support decompressing lz4 frame data.
  • The Azure Blob Storage and Google Cloud Storage archive destinations’ prefix fields support template syntax.
  • The Splunk HEC destination has a custom environment variable.
  • The sample processor has an optional group_by parameter.

Fixes

  • The Datadog Logs destination’s default compression has been updated to zstd, which matches Datadog Agent’s default compression.
  • The Amazon S3, Google Cloud Storage, and Azure Blob Storage destinations have been fixed to resolve log timestamps correctly.
  • The custom OCSF mapper’s performance has been improved.
  • The filter processor has flag logic enabled to pass events to the next processor.