Overview

The Sensitive Data Scanner processor scans logs or tracesContact your account manager to request access. to detect and redact or hash sensitive information such as PII, PCI, and custom sensitive data. You can pick from Datadog’s library of predefined rules, or input custom Regex rules to scan for sensitive data.

You can set up the pipeline and processor in the UI, API, or Terraform.

See Best practices to optimize performance for tips on reducing resource usage.

Set up the processor in the UI

To set up the processor:

1. Define a filter query. See Logs Search Syntax or APM Query Syntax for more information.
   • Only events matching the filter are scanned and processed.
   • All events, regardless of whether they match the filter query, are sent to the next step in the pipeline.
2. Click Add Scanning Rule.
3. Select one of the following:

Delete a rule

To delete a rule in the Sensitive Data Scanner:

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the Sensitive Data Scanner processor to expand it.
4. Click Manage Scanning Rules.
5. Select the rule you want to delete.
6. Click Delete.

Path notation example

For this log structure example:

{
    "outer_key": {
        "inner_key": "inner_value",
        "a": {
            "double_inner_key": "double_inner_value",
            "b": "b value"
        },
        "c": "c value"
    },
    "d": "d value"
}

Follow these reference rules:

• Use outer_key.inner_key to reference the key with the value inner_value.
• Use outer_key.a.double_inner_key to reference the key with the value double_inner_value.

To specify a nested field with a literal . in the attribute key, wrap the key in escaped quotes in the search query. For example, the search query "service.status":disabled matches the event {"service.status": "disabled"}.

Set up the processor using Terraform

You can use the Datadog Observability Pipeline Terraform resource to set up a pipeline with the Sensitive Data Scanner processor. To add a rule to the Sensitive Data Scanner processor using Terraform:

1. Use the Datadog Sensitive Data Scanner Standard Pattern data source to retrieve the rule ID of the Sensitive Data Scanner library rule.

   Copy

   data "datadog_sensitive_data_scanner_standard_pattern" "<RULE_IDENTIFIER>" {
     filter = "<RULE_NAME>"
   }
      

   Replace the placeholders:

   • <RULE_IDENTIFIER> with a name to use when you later set up the Sensitive Data Scanner processor in the Observability Pipeline resource.
   • <RULE_NAME> with the exact name of the rule. See Library Rules for the full list of rules.

   For example, if you want to use the AWS Access Key ID Scanner, configure the data source as follows:

   Copy

   data "datadog_sensitive_data_scanner_standard_pattern" "aws_access_key" {
     filter = "AWS Access Key ID Scanner"
   }
      

   See the full configuration example on how to add data sources for multiple rules.

2. Add a rule block in your Observability Pipeline resource for the library rule.

   Copy

   ...
     sensitive_data_scanner {
       rule {
         name = "<YOUR_RULE_NAME>"
         tags = []
         on_match {
           redact {
             replace = "***"
           }
         }
         pattern {
           library {
             id                       = data.datadog_sensitive_data_scanner_standard_pattern.<RULE_IDENTIFIER>.id
             use_recommended_keywords = true
           }
         }
         scope {
           all = true
         }
       }
     }
      

   Replace the placeholders:

   • <YOUR_RULE_NAME> with a name for the rule. This name is shown in the Pipelines UI.
   • <RULE_IDENTIFIER> with the rule identifier you used in the data source in step 1.

   For example, if you use the AWS Access Key ID Scanner data source from step 1, configure the rule block as follows:

   Copy

   ...
     sensitive_data_scanner {
       rule {
         name = "Redact AWS Access Key IDs"
         tags = []
         on_match {
           redact {
             replace = "***"
           }
         }
         pattern {
           library {
             id                       = data.datadog_sensitive_data_scanner_standard_pattern.aws_access_key.id
             use_recommended_keywords = true
           }
         }
         scope {
           all = true
         }
       }
     }
      

   See the full configuration example on how to add multiple rules.

3. Repeat steps 1 and 2 for all library rules you want to add.

Full configuration example

If you want to use the Sensitive Data Scanner processor to scan for AWS Access Key IDs and US Social Security Numbers, and redact them by replacing them with the string ***:

1. Use the Datadog Sensitive Data Scanner Standard Pattern data source to retrieve the rule IDs for the AWS Access Key ID Scanner and the US Social Security Number Scanner.
2. In your Datadog Observability Pipeline resource’s Sensitive Data Scanner processor, use the Sensitive Data Scanner rules defined in the data sources.

data "datadog_sensitive_data_scanner_standard_pattern" "aws_access_key" {
  filter = "AWS Access Key ID Scanner"
}
data "datadog_sensitive_data_scanner_standard_pattern" "us_ssn" {
  filter = "US Social Security Number Scanner"
}

resource "datadog_observability_pipeline" "sensitive_data_pipeline" {
  name = "Sensitive Data Pipeline"

  config {
    source {
      id = "source-0"
      datadog_agent {}
    }

    processor_group {
      display_name = "Processors"
      enabled      = true
      id           = "group-0"
      include      = "*"
      inputs       = ["source-0"]

      processor {
        display_name = "Sensitive Data Scanner"
        enabled      = true
        id           = "processor-sds-0"
        include      = "*"

        sensitive_data_scanner {
          rule {
            name = "Redact AWS Access Key IDs"
            tags = []
            on_match {
              redact {
                replace = "***"
              }
            }
            pattern {
              library {
                id                       = data.datadog_sensitive_data_scanner_standard_pattern.aws_access_key.id
                use_recommended_keywords = true
              }
            }
            scope {
              all = true
            }
          }
          rule {
            name = "Redact US SSNs"
            tags = []
            on_match {
              redact {
                replace = "***"
              }
            }
            pattern {
              library {
                id                       = data.datadog_sensitive_data_scanner_standard_pattern.us_ssn.id
                use_recommended_keywords = true
              }
            }
            scope {
              all = true
            }
          }
        }
      }
    }

    destination {
      id     = "destination-0"
      inputs = ["group-0"]
      datadog_logs {}
    }
  }
}

Best practices to optimize performance

The Sensitive Data Scanner processor is CPU intensive. Use the following best practices to optimize performance.

View scanning rule usage with the Observability Pipelines Overview dashboard

Observability Pipelines includes an out-of-the-box Observability Pipelines Overview dashboard with a Sensitive data found by Observability Pipelines section. Use the widgets in that section to see which scanning rules are matching data.

1. Navigate to Dashboards > Observability Pipelines Overview.
2. Use the template variables (pipeline_id, host, worker_uuid, component_type, component_kind, component_id) at the top of the dashboard to scope the view to a specific pipeline or Worker.
3. Use the time selector to scope to a wider time frame.

Use the following widgets to evaluate your Sensitive Data Scanner processors’ scanning rule usage:

• Logs containing sensitive data per scanning rule: Lists each rule by name (for example, visa_card_scanner_1x16_1x19_digits or redact_ipv4) with the number of matches over the selected time frame. Rules with high counts are actively matching data. This is the primary widget to see which rules are in use.
• Total count of logs containing sensitive data: Shows the total volume of sensitive data matched across all rules.
• Logs containing sensitive data by Pipeline: Shows matching logs that contain sensitive data. You can scope matches down by pipeline_id, which helps you see whether logs containing sensitive data is found in all pipelines or only in specific pipelines.
• Logs containing sensitive data per host: Breaks down sensitive data matches by Worker host. Use this widget to confirm coverage across your deployment.
• Patterns containing sensitive information and List of logs containing sensitive data: Shows the log patterns and sample events where sensitive data was found.

After you identify rules with no matches over a representative time frame, confirm they are not needed and remove them. See Delete a rule.

Note: A rule with zero matches means the rule did not match in the selected time frame, not that the rule is invalid.

Only enable rules you need

Rules that are enabled but not used consume unnecessary resources. Check the Sensitive Data Scanner processor to view how many matches each rule has had over the past 24 hours.

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the Sensitive Data Scanner processor to expand it.
4. Click View Scanning Rules to open the side panel and see Matches in the last 24 hours for each rule.

See Delete a rule to delete an unused rule.

Only scan the events and fields that need to be scanned for sensitive data

The time it takes the Sensitive Data Scanner to scan an event roughly scales with the size of the event. To optimize processor performance:

• If you know the types of events you want to scan, define a processor query that only sends the events you want to the processor.

• Reduce scanning time by targeting specific event attributes for scanning or excluding event attributes from being scanned. See the Define rule target and conditions step in Set up the processor.

Evaluate and benchmark performance optimizations

Use the pipelines.component_latency_seconds metric to:

• Benchmark processor performance when you add a rule
• Evaluate performance after making optimization changes, such as reducing the number of fields being scanned and removing unused rules

To view the pipelines.component_latency_seconds metric:

1. Navigate to Metrics Explorer.
2. In the metric field, enter pipelines.component_latency_seconds.
3. In the from field, enter the tag component_id:<COMPONENT_ID>, where <COMPONENT_ID> is the ID for your Sensitive Data Scanner processor.

Note: pipelines.component_latency_seconds is a distribution metric so you must enable percentiles for that metric. See Enabling advanced query functionality for instructions.

Health metrics

For component metrics and processor buffer metrics emitted by all processors, see the Pipelines Usage Metrics documentation.

Sensitive Data Scanner metrics

• Use the component_id tag to filter or group by individual components.
• The component_type tag is sensitive_data_scanner for Sensitive Data Scanner processor metrics.

pipelines.sds_rule_matched_total

Description: The number of events that matched a Sensitive Data Scanner rule. Tagged with the matching rule name.

Metric type: count

pipelines.scanned_events

Description: The number of events scanned by the Sensitive Data Scanner engine.

Metric type: count

pipelines.scanning.match_count

Description: The number of matches found by the Sensitive Data Scanner.

Metric type: count

pipelines.scanning.suppressed_match_count

Description: The number of matches suppressed by the Sensitive Data Scanner.

Metric type: count

pipelines.scanning.duration

Description: Accumulated wall-clock time, in seconds, spent scanning events. Use this metric to benchmark processor performance and evaluate optimizations.

Metric type: count

pipelines.scanning.cpu_duration

Description: Accumulated CPU time, in seconds, spent scanning events.

Metric type: count

pipelines.scanner.total_count

Description: The number of Sensitive Data Scanner processors currently running.

Metric type: gauge

pipelines.scanner.total_regexes

Description: The number of regexes held across all Sensitive Data Scanners.

Metric type: gauge

Further reading

추가 유용한 문서, 링크 및 기사:

Writing Effective Grok Parsing Rules with Regular ExpressionsGUIDE Route OTel data from AI apps to ClickHouse and Datadog using Observability PipelinesBLOG