--- title: Array Map Processor description: >- Learn how to transform each element of a source array, and collect results into a target array, in your log processors. breadcrumbs: Docs > Log Management > Log Configuration > Processors > Array Map Processor --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Array Map Processor ## Overview{% #overview %} Use the array map processor to apply a sequence of sub-processors to each element of a source array, producing a target array. Datadog supports the following sub-processors: - [Attribute Remapper](https://docs.datadoghq.com/logs/log_configuration/processors/remapper.md) - [String Builder](https://docs.datadoghq.com/logs/log_configuration/processors/string_builder_processor.md) - [Arithmetic Processor](https://docs.datadoghq.com/logs/log_configuration/processors/arithmetic_processor.md) - [Category Processor](https://docs.datadoghq.com/logs/log_configuration/processors/category_processor.md) {% alert level="info" %} - If `source` and `target` are the same attribute, the processor transforms elements in place. - If the `target` array already exists, the process overwrites attributes on existing elements. - Processing is limited to the first 50 elements of the source array. {% /alert %} ## Setup{% #setup %} Define the array map processor on the [Pipelines page](https://app.datadoghq.com/logs/pipelines): 1. Under Source array attribute, enter the path to the array attribute to iterate over. 1. Select or clear the Preserve source array checkbox to keep or remove the original source array after processing. 1. Under Target array attribute, enter the path to where to write the output array. 1. Add one or more sub-processors to apply to each element. {% image source="https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-processor.dd6519b051036003cbd1f7dd5ab642dd.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-processor.dd6519b051036003cbd1f7dd5ab642dd.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Screenshot of the Array Map Processor configuration panel" /%} ## Before and after state of logs{% #before-and-after-state-of-logs %} {% collapsible-section %} #### Example: Normalizing an array of network connections **Before:** ```json { "connections": [ { "src_ip": "10.0.0.1", "dst_port": 443, "proto": "tcp" }, { "src_ip": "10.0.0.2", "dst_port": 22, "proto": "tcp" } ] } ``` **Array Map Processor** Create an Array Map Processor with source `connections` and target `network.connections`. Add an Attribute Remapper sub-processor mapping `$sourceElem.src_ip` to `$targetElem.source`, and a String Builder sub-processor with template `%{$sourceElem.proto}/%{$sourceElem.dst_port}` writing to `$targetElem.service`. **After:** ```json { "connections": [...], "network": { "connections": [ {"source": "10.0.0.1", "service": "tcp/443"}, {"source": "10.0.0.2", "service": "tcp/22"} ] } } ``` {% /collapsible-section %} ## API{% #api %} Use the [Datadog Log Pipeline API endpoint](https://docs.datadoghq.com/api/v1/logs-pipelines.md) with the following Array Map processor JSON payload: ```json { "type": "array-map-processor", "name": "", "is_enabled": true, "source": "", "target": "", "preserve_source": true, "processors": [ {"type": "", ...}, {"type": "", ...} ] } ``` | Parameter | Type | Required | Description | | ----------------- | ---------------- | -------- | -------------------------------------------------------------------------------------------- | | `type` | String | Yes | Type of the processor. | | `name` | String | No | Name of the processor. | | `is_enabled` | Boolean | No | If the processor is enabled or not. Default: `false`. | | `source` | String | Yes | Name of the source array attribute. | | `target` | String | Yes | Name of the target array attribute. If equal to `source`, elements are transformed in place. | | `preserve_source` | Boolean | No | If the source array should be preserved after processing. Default: `true`. | | `processors` | Array of Objects | Yes | Sub-processors applied to each element, in order. At least one is required. | ## Sub-processors{% #sub-processors %} The following rules apply when defining sub-processors: - Inside each sub-processor, use: - `$sourceElem` to reference the current input element. - `$targetElem` to write to the current output element. - If an attribute does not start with `$sourceElem`, the processor reads from the parent log instead of the element itself. - `$sourceElem.`/`$targetElem.` refer to a nested attribute in the element object, while `$sourceElem`/`$targetElem` refer to a primitive element (for example, string, integer, double, Boolean). - All targets must start with `$targetElem`. ### Attribute Remapper{% #attribute-remapper %} Remaps an existing field to a field in the output element. {% tab title="UI" %} {% image source="https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-attribute-remapper.523723698ca845ed6c3fa5b33092f921.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-attribute-remapper.523723698ca845ed6c3fa5b33092f921.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Screenshot of the Attribute Remapper sub-processor configuration" /%} **Example input:** ```json { "items": ["10.0.0.1", "10.0.0.2"] } ``` **Configuration steps:** - Source attributes: `$sourceElem` - Target attribute: `$targetElem.ip` - Preserve source: `enabled` **Result:** ```json { "items": [...], "out": [ {"ip": "10.0.0.1"}, {"ip": "10.0.0.2"} ] } ``` {% /tab %} {% tab title="API" %} ```json { "type": "attribute-remapper", "name": "Map primitive IP to object field", "sources": ["$sourceElem"], "target": "$targetElem.ip", "target_format": "auto", "preserve_source": true, "override_on_conflict": false } ``` | Parameter | Type | Required | Description | | ---------------------- | ---------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | String | Yes | Type of the sub-processor. | | `name` | String | No | Name of the sub-processor. | | `sources` | Array of strings | Yes | Array of source attributes. | | `target` | String | Yes | Target attribute. | | `target_format` | String | No | Defines if the attribute value should be cast to another type. Possible values: `auto`, `string`, `double` or `integer`. Default: `auto`. When set to `auto`, no cast is applied. | | `preserve_source` | Boolean | No | If the remapped source element should be preserved after processing. Default: `false`. | | `override_on_conflict` | Boolean | No | If the target element is already set, whether it should be overridden. Default: `false`. | {% /tab %} ### String Builder Processor{% #string-builder-processor %} Builds a new field in the output element from a template. {% tab title="UI" %} {% image source="https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-string-builder.b9f3f89c904ecf9e58269cc3734bd2a0.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-string-builder.b9f3f89c904ecf9e58269cc3734bd2a0.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Screenshot of the String Builder sub-processor configuration" /%} **Example input:** ```json { "region": "us-east-1", "items": [ {"name": "db-1"}, {"name": "db-2"} ] } ``` **Configuration steps:** - Target attribute: `$targetElem.fqdn` - Template: `%{$sourceElem.name}.%{region}` - Replace missing: disabled **Result:** ```json { "region": "us-east-1", "items": [...], "out": [ {"fqdn": "db-1.us-east-1"}, {"fqdn": "db-2.us-east-1"} ] } ``` {% /tab %} {% tab title="API" %} ```json { "type": "string-builder-processor", "name": "Build FQDN from element and parent attribute", "template": "%{$sourceElem.name}.%{region}", "target": "$targetElem.fqdn", "is_replace_missing": false } ``` | Parameter | Type | Required | Description | | -------------------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | String | Yes | Type of the sub-processor. | | `name` | String | No | Name of the sub-processor. | | `template` | String | Yes | A formula with one or more attributes and raw text. | | `target` | String | Yes | The name of the attribute that contains the result of the template. | | `is_replace_missing` | Boolean | No | If `true`, replaces all missing attributes of `template` with an empty string. If `false`, skips the operation for missing attributes. Default: `false`. | {% /tab %} ### Arithmetic Processor{% #arithmetic-processor %} Computes a numeric expression using element or log attributes and writes the result to the output element. {% tab title="UI" %} {% image source="https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-arithmetic.441501b2c30326acef1260eb8abdd835.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-arithmetic.441501b2c30326acef1260eb8abdd835.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Screenshot of the Arithmetic Processor sub-processor configuration" /%} **Example input:** ```json { "items": [ {"bytes": 1024}, {"bytes": 2048} ] } ``` **Configuration steps:** - Target attribute: `$targetElem.kb` - Formula: `$sourceElem.bytes / 1024` - Replace missing value: disabled **Result:** ```json { "items": [...], "out": [ {"kb": 1}, {"kb": 2} ] } ``` {% /tab %} {% tab title="API" %} ```json { "type": "arithmetic-processor", "name": "Convert bytes to KB", "expression": "$sourceElem.bytes / 1024", "target": "$targetElem.kb", "is_replace_missing": false } ``` | Parameter | Type | Required | Description | | -------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | String | Yes | Type of the sub-processor. | | `name` | String | No | Name of the sub-processor. | | `expression` | String | Yes | Arithmetic operation between one or more log attributes. | | `target` | String | Yes | Name of the attribute that contains the result of the arithmetic operation. | | `is_replace_missing` | Boolean | No | If `true`, replaces all missing attributes of `expression` with 0. If `false`, skips the operation if an attribute is missing. Default: `false`. | {% /tab %} ### Category Processor{% #category-processor %} Assigns a category to each output element based on a filter query matching element attributes. {% tab title="UI" %} {% image source="https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-category.1ac6eb2e84dff8a57efdcf362b8f6892.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/log_configuration/processor/array-map-category.1ac6eb2e84dff8a57efdcf362b8f6892.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Screenshot of the Category Processor sub-processor configuration" /%} **Example input:** ```json { "items": [ {"status": "critical"}, {"status": "warning"} ] } ``` **Configuration steps:** - Target attribute: `$targetElem.severity` - Categories: - All events that match `@$sourceElem.status:critical` are mapped to value `high` - All events that match `@$sourceElem.status:warning` are mapped to value `medium` **Result:** ```json { "items": [...], "out": [ {"severity": "high"}, {"severity": "medium"} ] } ``` {% /tab %} {% tab title="API" %} ```json { "type": "category-processor", "name": "Map status to severity", "target": "$targetElem.severity", "categories": [ {"filter": {"query": "@$sourceElem.status:critical"}, "name": "high"}, {"filter": {"query": "@$sourceElem.status:warning"}, "name": "medium"} ] } ``` | Parameter | Type | Required | Description | | ------------ | --------------- | -------- | ------------------------------------------------------------------------------------------------------ | | `type` | String | Yes | Must be `category-processor`. | | `name` | String | No | Name of the sub-processor. | | `categories` | Array of Object | Yes | Array of filters to match to a log, and their corresponding custom `name` values to assign to the log. | | `target` | String | Yes | Name of the target attribute whose value is defined by the matching category. | {% /tab %} ## Further reading{% #further-reading %} - [Pipelines Overview](https://docs.datadoghq.com/logs/log_configuration/pipelines.md)