For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/logs/log_configuration/processors/array_processor.md. A documentation index is available at /llms.txt.

Array Processor

Overview

Use the array processor to extract, aggregate, or transform values from JSON arrays within your logs.

Supported operations include:

  • Select value from a matching element
  • Compute the length of an array
  • Append a value to an array
  • Extract key-value pairs from an array

Each operation is configured through a dedicated processor.

Define the array processor on the Pipelines page.

Select value from matching element

Extract a specific value from an object inside an array when it matches a condition.

    Array processor - Select value from element

    Example input:

    {
  "httpRequest": {
    "headers": [
      {"name": "Referrer", "value": "https://example.com"},
      {"name": "Accept", "value": "application/json"}
    ]
  }
}

    Configuration steps:

    • Array path: httpRequest.headers
    • Condition: name:Referrer
    • Extract value of: value
    • Target attribute: referrer

    Result:

    {
  "httpRequest": {
    "headers": [...]
  },
  "referrer": "https://example.com"
}

    Use the Datadog Log Pipeline API endpoint with the following array processor JSON payload:

    {
  "type": "array-processor",
  "name": "Extract Referrer URL",
  "is_enabled": true,
  "operation" : {
    "type" : "select",
    "source": "httpRequest.headers",
    "target": "referrer",
    "filter": "name:Referrer",
    "value_to_extract": "value"
  }
}
    ParameterTypeRequiredDescription
    typeStringYesType of the processor.
    nameStringNoName of the processor.
    is_enabledBooleanNoWhether the processor is enabled. Default: false.
    operation.typeStringYesType of array processor operation.
    operation.sourceStringYesPath of the array you want to select from.
    operation.targetStringYesTarget attribute.
    operation.filterStringYesExpression to match an array element. The first matching element is selected.
    operation.value_to_extractStringYesAttribute to read in the selected element.

    Array length

    Compute the number of elements in an array.

      Array processor - Length

      Example input:

      {
  "tags": ["prod", "internal", "critical"]
}

      Configuration steps:

      • Array attribute: tags
      • Target attribute: tagCount

      Result:

      {
  "tags": ["prod", "internal", "critical"],
  "tagCount": 3
}

      Use the Datadog Log Pipeline API endpoint with the following array processor JSON payload:

      {
  "type": "array-processor",
  "name": "Compute number of tags",
  "is_enabled": true,
  "operation" : {
    "type" : "length",
    "source": "tags",
    "target": "tagCount"
  }
}
      ParameterTypeRequiredDescription
      typeStringYesType of the processor.
      nameStringNoName of the processor.
      is_enabledBooleanNoWhether the processor is enabled. Default: false.
      operation.typeStringYesType of array processor operation.
      operation.sourceStringYesPath of the array to extract the length of.
      operation.targetStringYesTarget attribute.

      Append to array

      Add an attribute value to the end of a target array attribute in the log.

      Note: If the target array attribute does not exist in the log, it is automatically created.

        Array processor - Append

        Example input:

        {
  "network": {
    "client": {
      "ip": "198.51.100.23"
    }
  },
  "sourceIps": ["203.0.113.1"]
}

        Configuration steps:

        • Attribute to append: "network.client.ip"
        • Array attribute to append to: sourceIps

        Result:

        {
  "network": {
    "client": {
      "ip": "198.51.100.23"
    }
  },
  "sourceIps": ["203.0.113.1", "198.51.100.23"]
}

        Use the Datadog Log Pipeline API endpoint with the following array processor JSON payload:

        {
  "type": "array-processor",
  "name": "Append client IP to sourceIps",
  "is_enabled": true,
  "operation" : {
    "type" : "append",
    "source": "network.client.ip",
    "target": "sourceIps"
  }
}
        ParameterTypeRequiredDescription
        typeStringYesType of the processor.
        nameStringNoName of the processor.
        is_enabledBooleanNoWhether the processor is enabled. Default: false.
        operation.typeStringYesType of array processor operation.
        operation.sourceStringYesAttribute to append.
        operation.targetStringYesArray attribute to append to.
        operation.preserve_sourceBooleanNoWhether to preserve the original source after remapping. Default: false.

        Extract key-value pairs

        Flatten an array of key-value objects into individual attributes, creating one attribute per element. Use this operation when a log carries data as a list of {key, value} objects, such as the request headers in AWS WAF and HTTP logs. Each entry becomes its own searchable, facetable attribute. Keys are handled automatically, even when they differ from one log to the next.

          Array processor - Extract key-value pairs

          Example input:

          {
  "httpRequest": {
    "headers": [
      {"name": "host", "value": "api.example.com"},
      {"name": "user-agent", "value": "curl/8.4.0"},
      {"name": "x-forwarded-for", "value": "203.0.113.7"}
    ]
  }
}

          Configuration steps:

          • Array path: httpRequest.headers
          • Key attribute: name
          • Value attribute: value
          • Target attribute: (optional) leave blank to add the extracted attributes at the root level of the log

          Result:

          {
  "httpRequest": {
    "headers": [...]
  },
  "host": "api.example.com",
  "user-agent": "curl/8.4.0",
  "x-forwarded-for": "203.0.113.7"
}

          Use the Datadog Log Pipeline API endpoint with the following array processor JSON payload:

          {
  "type": "array-processor",
  "name": "Extract HTTP headers",
  "is_enabled": true,
  "operation" : {
    "type" : "key-value",
    "source": "httpRequest.headers",
    "key_to_extract": "name",
    "value_to_extract": "value"
  }
}
          ParameterTypeRequiredDescription
          typeStringYesType of the processor.
          nameStringNoName of the processor.
          is_enabledBooleanNoWhether the processor is enabled. Default: false.
          operation.typeStringYesType of array processor operation.
          operation.sourceStringYesAttribute path of the array to extract key-value pairs from.
          operation.targetStringNoAttribute that receives the extracted key-value pairs. If not specified, the extracted attributes are added at the root level of the log.
          operation.key_to_extractStringYesKey of the attribute in each array element that holds the name to use for the extracted attribute.
          operation.value_to_extractStringYesKey of the attribute in each array element that holds the value to use for the extracted attribute.
          operation.override_on_conflictBooleanNoWhether to override the target attribute if it is already set. Default: false.

          Further reading

          Additional helpful documentation, links, and articles:

          Discover Datadog PipelinesDOCUMENTATION more more