Overview

The Sensitive Data Scanner processor scans logs to detect and redact or hash sensitive information such as PII, PCI, and custom sensitive data. You can pick from Datadog’s library of predefined rules, or input custom Regex rules to scan for sensitive data.

You can set up the pipeline and processor in the UI, API, or Terraform.

See Best practices to optimize performance for tips on reducing resource usage.

Set up the processor in the UI

To set up the processor:

1. Define a filter query. Only logs that match the specified filter query are scanned and processed. All logs are sent to the next step in the pipeline, regardless of whether they match the filter query. See Search Syntax for more information.
2. Click Add Scanning Rule.
3. Select one of the following:

Delete a rule

To delete a rule in the Sensitive Data Scanner:

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the Sensitive Data Scanner processor to expand it.
4. Click Manage Scanning Rules.
5. Select the rule you want to delete.
6. Click Delete.

Path notation example

For this log structure example:

{
    "outer_key": {
        "inner_key": "inner_value",
        "a": {
            "double_inner_key": "double_inner_value",
            "b": "b value"
        },
        "c": "c value"
    },
    "d": "d value"
}

Follow these reference rules:

• Use outer_key.inner_key to reference the key with the value inner_value.
• Use outer_key.a.double_inner_key to reference the key with the value double_inner_value.

To specify a nested field with a literal . in the attribute key, wrap the key in escaped quotes in the search query. For example, the search query "service.status":disabled matches the event {"service.status": "disabled"}.

Set up the processor using Terraform

You can use the Datadog Observability Pipeline Terraform resource to set up a pipeline with the Sensitive Data Scanner processor. To add a rule to the Sensitive Data Scanner processor using Terraform:

1. Use the Datadog Sensitive Data Scanner Standard Pattern data source to retrieve the rule ID of the Sensitive Data Scanner library rule.

   Copy

   data "datadog_sensitive_data_scanner_standard_pattern" "<RULE_IDENTIFIER>" {
     filter = "<RULE_NAME>"
   }
      

   Replace the placeholders:

   • <RULE_IDENTIFIER> with a name to use when you later set up the Sensitive Data Scanner processor in the Observability Pipeline resource.
   • <RULE_NAME> with the exact name of the rule. See Library Rules for the full list of rules.

   For example, if you want to use the AWS Access Key ID Scanner, configure the data source as follows:

   Copy

   data "datadog_sensitive_data_scanner_standard_pattern" "aws_access_key" {
     filter = "AWS Access Key ID Scanner"
   }
      

   See the full configuration example on how to add data sources for multiple rules.

2. Add a rule block in your Observability Pipeline resource for the library rule.

   Copy

   ...
     sensitive_data_scanner {
       rule {
         name = "<YOUR_RULE_NAME>"
         tags = []
         on_match {
           redact {
             replace = "***"
           }
         }
         pattern {
           library {
             id                       = data.datadog_sensitive_data_scanner_standard_pattern.<RULE_IDENTIFIER>.id
             use_recommended_keywords = true
           }
         }
         scope {
           all = true
         }
       }
     }
      

   Replace the placeholders:

   • <YOUR_RULE_NAME> with a name for the rule. This name is shown in the Pipelines UI.
   • <RULE_IDENTIFIER> with the rule identifier you used in the data source in step 1.

   For example, if you use the AWS Access Key ID Scanner data source from step 1, configure the rule block as follows:

   Copy

   ...
     sensitive_data_scanner {
       rule {
         name = "Redact AWS Access Key IDs"
         tags = []
         on_match {
           redact {
             replace = "***"
           }
         }
         pattern {
           library {
             id                       = data.datadog_sensitive_data_scanner_standard_pattern.aws_access_key.id
             use_recommended_keywords = true
           }
         }
         scope {
           all = true
         }
       }
     }
      

   See the full configuration example on how to add multiple rules.

3. Repeat steps 1 and 2 for all library rules you want to add.

Full configuration example

If you want to use the Sensitive Data Scanner processor to scan for AWS Access Key IDs and US Social Security Numbers, and redact them by replacing them with the string ***:

1. Use the Datadog Sensitive Data Scanner Standard Pattern data source to retrieve the rule IDs for the AWS Access Key ID Scanner and the US Social Security Number Scanner.
2. In your Datadog Observability Pipeline resource’s Sensitive Data Scanner processor, use the Sensitive Data Scanner rules defined in the data sources.

data "datadog_sensitive_data_scanner_standard_pattern" "aws_access_key" {
  filter = "AWS Access Key ID Scanner"
}
data "datadog_sensitive_data_scanner_standard_pattern" "us_ssn" {
  filter = "US Social Security Number Scanner"
}

resource "datadog_observability_pipeline" "sensitive_data_pipeline" {
  name = "Sensitive Data Pipeline"

  config {
    source {
      id = "source-0"
      datadog_agent {}
    }

    processor_group {
      display_name = "Processors"
      enabled      = true
      id           = "group-0"
      include      = "*"
      inputs       = ["source-0"]

      processor {
        display_name = "Sensitive Data Scanner"
        enabled      = true
        id           = "processor-sds-0"
        include      = "*"

        sensitive_data_scanner {
          rule {
            name = "Redact AWS Access Key IDs"
            tags = []
            on_match {
              redact {
                replace = "***"
              }
            }
            pattern {
              library {
                id                       = data.datadog_sensitive_data_scanner_standard_pattern.aws_access_key.id
                use_recommended_keywords = true
              }
            }
            scope {
              all = true
            }
          }
          rule {
            name = "Redact US SSNs"
            tags = []
            on_match {
              redact {
                replace = "***"
              }
            }
            pattern {
              library {
                id                       = data.datadog_sensitive_data_scanner_standard_pattern.us_ssn.id
                use_recommended_keywords = true
              }
            }
            scope {
              all = true
            }
          }
        }
      }
    }

    destination {
      id     = "destination-0"
      inputs = ["group-0"]
      datadog_logs {}
    }
  }
}

Best practices to optimize performance

The Sensitive Data Scanner processor is CPU intensive. Use the following best practices to optimize performance.

Only enable rules you need

Rules that are enabled but not used consume unnecessary resources. Check the Sensitive Data Scanner processor to view how many matches each rule has had over the past 24 hours.

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the Sensitive Data Scanner processor to expand it.
4. Click View Scanning Rules to open the side panel and see Matches in the last 24 hours for each rule.

See Delete a rule to delete an unused rule.

Only scan the events and fields that need to be scanned for sensitive data

The time it takes the Sensitive Data Scanner to scan an event roughly scales with the size of the event. To optimize processor performance:

• If you know the types of events you want to scan, define a processor query that only sends the events you want to the processor.

• Reduce scanning time by targeting specific event attributes for scanning or excluding event attributes from being scanned. See the Define rule target and conditions step in Set up the processor.

Evaluate and benchmark performance optimizations

Use the pipelines.component_latency_seconds metric to:

• Benchmark processor performance when you add a rule
• Evaluate performance after making optimization changes, such as reducing the number of fields being scanned and removing unused rules

To view the pipelines.component_latency_seconds metric:

1. Navigate to Metrics Explorer.
2. In the metric field, enter pipelines.component_latency_seconds.
3. In the from field, enter the tag component_id:<COMPONENT_ID>, where <COMPONENT_ID> is the ID for your Sensitive Data Scanner processor.

Note: pipelines.component_latency_seconds is a distribution metric so you must enable percentiles for that metric. See Enabling advanced query functionality for instructions.

Further reading

Additional helpful documentation, links, and articles:

Writing Effective Grok Parsing Rules with Regular ExpressionsGUIDE