Use this processor to enrich your logs with information from a reference table, which could be a local file or database.

To set up the enrichment table processor:

1. Define a filter query. Only logs that match the specified filter query are processed. All logs, regardless of whether they do or do not match the filter query, are sent to the next step in the pipeline.
2. Enter the source attribute of the log. The source attribute’s value is what you want to find in the reference table.
3. Enter the target attribute. The target attribute’s value stores, as a JSON object, the information found in the reference table.
4. Select the type of reference table you want to use, File or GeoIP.
   • For the File type:
     1. Enter the file path.
     2. Enter the column name. The column name in the enrichment table is used for matching the source attribute value. See the Enrichment file example.
        Note: If you are installing the Worker in Kubernetes, see Referencing files in Kubernetes for information on how to reference the file.
   • For the GeoIP type, enter the GeoIP path.

Enrichment file example

For this example, merchant_id is used as the source attribute and merchant_info as the target attribute.

This is the example reference table that the enrichment processor uses:

merch_id is set as the column name the processor uses to find the source attribute’s value. Note: The source attribute’s value does not have to match the column name.

If the enrichment processor receives a log with "merchant_id":"536":

• The processor looks for the value 536 in the reference table’s merch_id column.
• After it finds the value, it adds the entire row of information from the reference table to the merchant_info attribute as a JSON object:

merchant_info {
    "merchant_name":"Cindy's Couches",
    "city":"Boulder",
    "state":"Colorado"
}

Filter query syntax

Each processor has a corresponding filter query in their fields. Processors only process logs that match their filter query. And for all processors except the filter processor, logs that do not match the query are sent to the next step of the pipeline. For the filter processor, logs that do not match the query are dropped.

For any attribute, tag, or key:value pair that is not a reserved attribute, your query must start with @. Conversely, to filter reserved attributes, you do not need to append @ in front of your filter query.

For example, to filter out and drop status:info logs, your filter can be set as NOT (status:info). To filter out and drop system-status:info, your filter must be set as NOT (@system-status:info).

Filter query examples:

• NOT (status:debug): This filters for only logs that do not have the status DEBUG.
• status:ok service:flask-web-app: This filters for all logs with the status OK from your flask-web-app service.
  • This query can also be written as: status:ok AND service:flask-web-app.
• host:COMP-A9JNGYK OR host:COMP-J58KAS: This filter query only matches logs from the labeled hosts.
• @user.status:inactive: This filters for logs with the status inactive nested under the user attribute.

Learn more about writing filter queries in Datadog’s Log Search Syntax.