Lookup Processor

Docs > Log Management > Log Configuration > Processors > Lookup Processor

Overview

Use the lookup processor to define a mapping between a log attribute and a human readable value saved in a Reference Table or the processors mapping table.

For example, you can use the lookup processor to map an internal service ID into a human readable service name. Alternatively, you can use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

Use cases

The Lookup Processor is typically used to transform an attribute into a human-readable attribute. It is commonly used for:

Mapping a service ID to a human-readable service name
Mapping a user ID to a username
Mapping a bit value to a human-readable attribute

Setup

The lookup processor performs the following actions:

Looks if the current log contains the source attribute.
Checks if the source attribute value exists in the mapping table.
- If it does, creates the target attribute with the corresponding value in the table.
- Optionally, if it does not find the value in the mapping table, it creates a target attribute with the default fallback value set in the fallbackValue field. You can manually enter a list of source_key,target_value pairs or upload a CSV file on the Manual Mapping tab.
  The size limit for the mapping table is 100Kb. This limit applies across all Lookup Processors on the platform. However, Reference Tables support larger file sizes.
- Optionally, if it does not find the value in the mapping table, it creates a target attribute with the value of the reference table. You can select a value for a Reference Table on the Reference Table tab.

Before and after state of logs

Example: Using the Lookup Processor to enrich logs with human-readable values

Before:

{
  "transaction": {
    "id": "tx_98765",
    "status_code": 2
  },
  "user": {
    "id": "42"
  }
}

Lookup Processor

Create a Lookup Processor with source attribute user.id and a lookup table containing users: 42 → alice.doe 77 → bob.smith

After processing:

{
  "transaction": {
    "id": "tx_98765",
    "status_code": 2
  },
  "user": {
    "id": "42",
    "username": "alice.doe"
  }
}

API

Use the Datadog Log Pipeline API endpoint with the following lookup processor JSON payload:

{
  "type": "lookup-processor",
  "name": "<PROCESSOR_NAME>",
  "is_enabled": true,
  "source": "<SOURCE_ATTRIBUTE>",
  "target": "<TARGET_ATTRIBUTE>",
  "lookup_table": ["key1,value1", "key2,value2"],
  "default_lookup": "<DEFAULT_TARGET_VALUE>"
}

Parameter	Type	Required	Description
`type`	String	Yes	Type of the processor.
`name`	String	No	Name of the processor.
`is_enabled`	Boolean	Yes	If the processor is enabled or not. Default: `false`.
`source`	String	Yes	Source attribute used to perform the lookup.
`target`	String	Yes	Name of the attribute that contains the corresponding value in the mapping list or the `default_lookup` if not found in the mapping list.
`lookup_table`	Array of strings	Yes	Mapping table of values for the source attribute and their associated target attribute values, formatted as [ “source_key1,target_value1”, “source_key2,target_value2” ].
`default_lookup`	String	No	Value to set the target attribute if the source value is not found in the list.