Use this processor to set a limit on the number of logs sent within a specific time window. For example, you can set a limit so that only 100 logs are sent per second. Setting a rate limit can help you catch any spikes in log ingestion and prevent unexpected billing costs.

To set up the processor:

1. Define a filter query. Only logs that match the specified filter query are processed. All matched logs get throttled. Logs that are sent within the throttle limit and logs that do not match the filter are sent to the next step. Logs sent after the throttle limit has been reached, are dropped.
2. Set the throttling rate. This is the number of events allowed for a given bucket during the set time window. Note: This rate limit is applied on a per-worker level. If you scale the number of workers up or down, you may want to adjust the processor rate limit accordingly. You can update the rate limit programmatically using the Observability Pipelines API.
3. Set the time window.
4. Optionally, click Add Field if you want to group by a field.

Overview

Use this processor to set a limit on the number of logs sent within a specific time window. For example, you can set a limit so that only 100 logs are sent per second. Setting a rate limit can help you catch any spikes in log ingestion and prevent unexpected billing costs.

Setup

To set up the processor:

1. Define a filter query.
   • Only logs that match the specified filter query are processed.
   • All matched logs get throttled. Logs that are sent within the throttle limit and logs that do not match the filter are sent to the next step. Logs sent after the throttle limit has been reached, are dropped.
2. Set the throttling rate. This is the number of events allowed for a given bucket during the set time window.
   • Note: This rate limit is applied on a per-worker level. If you scale the number of workers up or down, you may want to adjust the processor rate limit accordingly. You can update the rate limit programmatically using the Observability Pipelines API.
3. Set the time window.
4. Optionally, click Add Field if you want to group by a field.

Filter query syntax

Each processor has a corresponding filter query in their fields. Processors only process logs that match their filter query. And for all processors except the Filter processor, logs that do not match the query are sent to the next step of the pipeline. For the Filter processor, logs that do not match the query are dropped.

The following are logs filter query examples:

• NOT (status:debug): This filters for logs that do not have the status DEBUG.
• status:ok service:flask-web-app: This filters for all logs with the status OK from your flask-web-app service.
  • This query can also be written as: status:ok AND service:flask-web-app.
• host:COMP-A9JNGYK OR host:COMP-J58KAS: This filter query only matches logs from the labeled hosts.
• user.status:inactive: This filters for logs with the status inactive nested under the user attribute.
• http.status:[200 TO 299] or http.status:{300 TO 399}: These two filters represent the syntax to query a range for http.status. Ranges can be used across any attribute.

Learn more about writing filter queries in Observability Pipelines Search Syntax.

How the Throttle processor works

The Throttle processor sets a rate limit on the number of logs sent within a specified time window. While similar to the Quota processor, the main difference between the Throttle and Quota processors is that the Quota processor’s time window is fixed at 24 hours and cannot be changed, while the Throttle processor’s time window can be configured. Since the Throttle processor’s time window is configurable, the processor has a capacity replenishment rate based on the throttling rate and time window you set. See Capacity replenishment rate for more information.

The following table compares the Throttle processor with the Quota processor:

Initial capacity

When the Throttle processor is enabled, the number of logs the processor allows through immediately is based on the configured Throttling Rate. For example, if the Throttling Rate is set to 1000 events over 60 seconds, and 5,000 events arrive the moment the processor is enabled:

• The processor allows an initial capacity of 1,000 events to pass through.
• The remaining 4,000 events are dropped.
• This initial behavior is identical to a Quota processor’s.

Capacity replenishment rate

The Throttle processor uses a generic cell rate algorithm, which enables a steady rate of events to pass through. The replenishment rate is based on the settings of your Throttle processor and allows a certain number of events to pass through per second. This rate can be calculated as follows:

$$\text"Throttle rate" / \text"Time window (in seconds)"$$

Example

If you use the following processor settings:

• Throttling rate = 1000 events
• Time window = 60 minutes (3600 seconds)

The capacity replenishment rate is:

$$\text"1000 events" / \text"60 minutes" ≈ \text"17 events"/ \text"minute" ≈ \text"0.28 events"/ \text"second"$$

If T is the time when the processor is enabled and the processor receives 5000 events at that time, the number of events that the processor allows through based on T is as follows:

• T + 0 minutes (when the processor is enabled):
  • 1000 events processed.
  • 4000 events dropped.
• T + 1 minute: ~17 events can be processed
• T + 2 minutes: ~17 events can be processed
• …the processor continues processing events at a steady rate of ~17 events per minute and dropping the rest until the next minute.

Note: The replenishment rate determines the maximum throughput after the initial capacity. You can adjust the throttling rate for a higher or lower throughput if needed.