Overview

Use the array map processor to apply a sequence of sub-processors to each element of a source array, producing a target array. Datadog supports the following sub-processors:

• Attribute Remapper
• String Builder
• Arithmetic Processor
• Category Processor

Setup

Define the array map processor on the Pipelines page:

1. Under Source array attribute, enter the path to the array attribute to iterate over.
2. Select or clear the Preserve source array checkbox to keep or remove the original source array after processing.
3. Under Target array attribute, enter the path to where to write the output array.
4. Add one or more sub-processors to apply to each element.

Before and after state of logs

{
  "connections": [
    { "src_ip": "10.0.0.1", "dst_port": 443, "proto": "tcp" },
    { "src_ip": "10.0.0.2", "dst_port": 22,  "proto": "tcp" }
  ]
}

{
  "connections": [...],
  "network": {
    "connections": [
      {"source": "10.0.0.1", "service": "tcp/443"},
      {"source": "10.0.0.2", "service": "tcp/22"}
    ]
  }
}

API

Use the Datadog Log Pipeline API endpoint with the following Array Map processor JSON payload:

{
  "type": "array-map-processor",
  "name": "<PROCESSOR_NAME>",
  "is_enabled": true,
  "source": "<SOURCE_ARRAY_ATTRIBUTE>",
  "target": "<TARGET_ARRAY_ATTRIBUTE>",
  "preserve_source": true,
  "processors": [
    {"type": "<SUB_PROCESSOR_TYPE>", ...},
    {"type": "<SUB_PROCESSOR_TYPE>", ...}
  ]
}

Sub-processors

The following rules apply when defining sub-processors:

• Inside each sub-processor, use:
  • $sourceElem to reference the current input element.
  • $targetElem to write to the current output element.
• If an attribute does not start with $sourceElem, the processor reads from the parent log instead of the element itself.
• $sourceElem.<field>/$targetElem.<field> refer to a nested attribute in the element object, while $sourceElem/$targetElem refer to a primitive element (for example, string, integer, double, Boolean).
• All targets must start with $targetElem.

Attribute Remapper

Remaps an existing field to a field in the output element.

Example input:

{
  "items": ["10.0.0.1", "10.0.0.2"]
}

Configuration steps:

• Source attributes: $sourceElem
• Target attribute: $targetElem.ip
• Preserve source: enabled

Result:

{
  "items": [...],
  "out": [
    {"ip": "10.0.0.1"},
    {"ip": "10.0.0.2"}
  ]
}

{
  "type": "attribute-remapper",
  "name": "Map primitive IP to object field",
  "sources": ["$sourceElem"],
  "target": "$targetElem.ip",
  "target_format": "auto",
  "preserve_source": true,
  "override_on_conflict": false
}

Parameter	Type	Required	Description
type	String	Yes	Type of the sub-processor.
name	String	No	Name of the sub-processor.
sources	Array of strings	Yes	Array of source attributes.
target	String	Yes	Target attribute.
target_format	String	No	Defines if the attribute value should be cast to another type. Possible values: auto, string, double or integer. Default: auto. When set to auto, no cast is applied.
preserve_source	Boolean	No	If the remapped source element should be preserved after processing. Default: false.
override_on_conflict	Boolean	No	If the target element is already set, whether it should be overridden. Default: false.

String Builder Processor

Builds a new field in the output element from a template.

Example input:

{
  "region": "us-east-1",
  "items": [
    {"name": "db-1"},
    {"name": "db-2"}
  ]
}

Configuration steps:

• Target attribute: $targetElem.fqdn
• Template: %{$sourceElem.name}.%{region}
• Replace missing: disabled

Result:

{
  "region": "us-east-1",
  "items": [...],
  "out": [
    {"fqdn": "db-1.us-east-1"},
    {"fqdn": "db-2.us-east-1"}
  ]
}

{
  "type": "string-builder-processor",
  "name": "Build FQDN from element and parent attribute",
  "template": "%{$sourceElem.name}.%{region}",
  "target": "$targetElem.fqdn",
  "is_replace_missing": false
}

Parameter	Type	Required	Description
type	String	Yes	Type of the sub-processor.
name	String	No	Name of the sub-processor.
template	String	Yes	A formula with one or more attributes and raw text.
target	String	Yes	The name of the attribute that contains the result of the template.
is_replace_missing	Boolean	No	If true, replaces all missing attributes of template with an empty string. If false, skips the operation for missing attributes. Default: false.

Arithmetic Processor

Computes a numeric expression using element or log attributes and writes the result to the output element.

Example input:

{
  "items": [
    {"bytes": 1024},
    {"bytes": 2048}
  ]
}

Configuration steps:

• Target attribute: $targetElem.kb
• Formula: $sourceElem.bytes / 1024
• Replace missing value: disabled

Result:

{
  "items": [...],
  "out": [
    {"kb": 1},
    {"kb": 2}
  ]
}

{
  "type": "arithmetic-processor",
  "name": "Convert bytes to KB",
  "expression": "$sourceElem.bytes / 1024",
  "target": "$targetElem.kb",
  "is_replace_missing": false
}

Parameter	Type	Required	Description
type	String	Yes	Type of the sub-processor.
name	String	No	Name of the sub-processor.
expression	String	Yes	Arithmetic operation between one or more log attributes.
target	String	Yes	Name of the attribute that contains the result of the arithmetic operation.
is_replace_missing	Boolean	No	If true, replaces all missing attributes of expression with 0. If false, skips the operation if an attribute is missing. Default: false.

Category Processor

Assigns a category to each output element based on a filter query matching element attributes.

Example input:

{
  "items": [
    {"status": "critical"},
    {"status": "warning"}
  ]
}

Configuration steps:

• Target attribute: $targetElem.severity
• Categories:
  • All events that match @$sourceElem.status:critical are mapped to value high
  • All events that match @$sourceElem.status:warning are mapped to value medium

Result:

{
  "items": [...],
  "out": [
    {"severity": "high"},
    {"severity": "medium"}
  ]
}

{
  "type": "category-processor",
  "name": "Map status to severity",
  "target": "$targetElem.severity",
  "categories": [
    {"filter": {"query": "@$sourceElem.status:critical"}, "name": "high"},
    {"filter": {"query": "@$sourceElem.status:warning"},  "name": "medium"}
  ]
}

Parameter	Type	Required	Description
type	String	Yes	Must be category-processor.
name	String	No	Name of the sub-processor.
categories	Array of Object	Yes	Array of filters to match to a log, and their corresponding custom name values to assign to the log.
target	String	Yes	Name of the target attribute whose value is defined by the matching category.

Further reading

Additional helpful documentation, links, and articles:

Pipelines OverviewDOCUMENTATION