--- title: Sensitive Data Scanner Processor description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > Observability Pipelines > Processors > Sensitive Data Scanner Processor --- # Sensitive Data Scanner Processor {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). (). {% /alert %} {% /callout %} Available for: {% icon name="icon-logs" /%} Logs ## Overview{% #overview %} The Sensitive Data Scanner processor scans logs to detect and redact or hash sensitive information such as PII, PCI, and custom sensitive data. You can pick from Datadog's library of predefined rules, or input custom Regex rules to scan for sensitive data. You can set up the pipeline and processor in the UI, [API](https://docs.datadoghq.com/api/latest/observability-pipelines/#create-a-new-pipeline), or Terraform. ## Set up the processor in the UI{% #set-up-the-processor-in-the-ui %} To set up the processor: 1. Define a filter query. Only logs that match the specified filter query are scanned and processed. All logs are sent to the next step in the pipeline, regardless of whether they match the filter query. See [Search Syntax](https://docs.datadoghq.com/observability_pipelines/search_syntax/logs/) for more information. 1. Click **Add Scanning Rule**. 1. Select one of the following: {% tab title="Library rules" %} 1. In the dropdown menu, select the library rule you want to use. 1. Recommended keywords are automatically added based on the library rule selected. After the scanning rule has been added, you can add additional keywords or remove recommended keywords. 1. In the **Define rule target and action** section, select if you want to scan the **Entire Event**, **Specific Attributes**, or **Exclude Attributes** in the dropdown menu. - If you are scanning the entire event, you can optionally exclude specific attributes from getting scanned. Use path notation (`outer_key.inner_key`) to access nested keys. For specified attributes with nested data, all nested data is excluded. - If you are scanning specific attributes, specify which attributes you want to scan. Use path notation (`outer_key.inner_key`) to access nested keys. For specified attributes with nested data, all nested data is scanned. 1. For **Define actions on match**, select the action you want to take for the matched information. **Note**: Redaction, partial redaction, and hashing are all irreversible actions. - **Redact**: Replaces all matching values with the text you specify in the **Replacement text** field. - **Partially Redact**: Replaces a specified portion of all matched data. In the **Redact** section, specify the number of characters you want to redact and which part of the matched data to redact. - **Hash**: Replaces all matched data with a unique identifier. The UTF-8 bytes of the match are hashed with the 64-bit fingerprint of FarmHash. 1. Optionally, click **Add Field** to add tags you want to associate with the matched events. 1. Add a name for the scanning rule. 1. Optionally, add a description for the rule. 1. Click **Save**. ### Add additional keywords{% #add-additional-keywords %} After adding scanning rules from the library, you can edit each rule separately and add additional keywords to the keyword dictionary. 1. Navigate to your [pipeline](https://app.datadoghq.com/observability-pipelines). 1. In the Sensitive Data Scanner processor with the rule you want to edit, click **Manage Scanning Rules**. 1. Toggle **Use recommended keywords** if you want the rule to use them. Otherwise, add your own keywords to the **Create keyword dictionary** field. You can also require that these keywords be within a specified number of characters of a match. By default, keywords must be within 30 characters before a matched value. 1. Click **Update**. {% /tab %} {% tab title="Custom rules" %} 1. In the **Define match conditions** section, specify the regex pattern to use for matching against events in the **Define the regex** field. See [Writing Effective Grok Parsing Rules with Regular Expressions](https://docs.datadoghq.com/logs/guide/regex_log_parsing/) for more information. Sensitive Data Scanner supports Perl Compatible Regular Expressions (PCRE), but the following patterns are not supported: - Backreferences and capturing sub-expressions (lookarounds) - Arbitrary zero-width assertions - Subroutine references and recursive patterns - Conditional patterns - Backtracking control verbs - The `\C` "single-byte" directive (which breaks UTF-8 sequences) - The `\R` newline match - The `\K` start of match reset directive - Callouts and embedded code - Atomic grouping and possessive quantifiers 1. Enter sample data in the **Add sample data** field to verify that your regex pattern is valid. 1. For **Create keyword dictionary**, add keywords to refine detection accuracy when matching regex conditions. For example, if you are scanning for a sixteen-digit Visa credit card number, you can add keywords like `visa`, `credit`, and `card`. You can also require that these keywords be within a specified number of characters of a match. By default, keywords must be within 30 characters before a matched value. 1. In the **Define rule target and action** section, select if you want to scan the **Entire Event**, **Specific Attributes**, or **Exclude Attributes** in the dropdown menu. - If you are scanning the entire event, you can optionally exclude specific attributes from getting scanned. Use path notation (`outer_key.inner_key`) to access nested keys. For specified attributes with nested data, all nested data is excluded. - If you are scanning specific attributes, specify which attributes you want to scan. Use path notation (`outer_key.inner_key`) to access nested keys. For specified attributes with nested data, all nested data is scanned. 1. For **Define actions on match**, select the action you want to take for the matched information. **Note**: Redaction, partial redaction, and hashing are all irreversible actions. - **Redact**: Replaces all matching values with the text you specify in the **Replacement text** field. - **Partially Redact**: Replaces a specified portion of all matched data. In the **Redact** section, specify the number of characters you want to redact and which part of the matched data to redact. - **Hash**: Replaces all matched data with a unique identifier. The UTF-8 bytes of the match is hashed with the 64-bit fingerprint of FarmHash. 1. Optionally, click **Add Field** to add tags you want to associate with the matched events. 1. Add a name for the scanning rule. 1. Optionally, add a description for the rule. 1. Click **Add Rule**. {% /tab %} ### Path notation example{% #path-notation-example %} For the following message structure: ```json { "outer_key": { "inner_key": "inner_value", "a": { "double_inner_key": "double_inner_value", "b": "b value" }, "c": "c value" }, "d": "d value" } ``` - Use `outer_key.inner_key` to refer to the key with the value `inner_value`. - Use `outer_key.inner_key.double_inner_key` to refer to the key with the value `double_inner_value`. ## Set up the processor using Terraform{% #set-up-the-processor-using-terraform %} You can use the [Datadog Observability Pipeline Terraform resource](https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/observability_pipeline) to set up a pipeline with the Sensitive Data Scanner processor. To add a rule to the Sensitive Data Scanner processor using Terraform: 1. Use the [Datadog Sensitive Data Scanner Standard Pattern](https://registry.terraform.io/providers/DataDog/datadog/latest/docs/data-sources/sensitive_data_scanner_standard_pattern) data source to retrieve the rule ID of the Sensitive Data Scanner [library rule](https://docs.datadoghq.com/security/sensitive_data_scanner/scanning_rules/library_rules/). ```terraform data "datadog_sensitive_data_scanner_standard_pattern" "" { filter = "" } ``` Replace the placeholders: - `` with a name to use when you later set up the Sensitive Data Scanner processor in the Observability Pipeline resource. - `` with the exact name of the rule. See [Library Rules](https://docs.datadoghq.com/security/sensitive_data_scanner/scanning_rules/library_rules/) for the full list of rules. For example, if you want to use the [AWS Access Key ID Scanner](https://docs.datadoghq.com/security/sensitive_data_scanner/scanning_rules/library_rules/?search=AWS+Access+Key+ID+Scanner), configure the data source as follows: ```terraform data "datadog_sensitive_data_scanner_standard_pattern" "aws_access_key" { filter = "AWS Access Key ID Scanner" } ``` See the full configuration example on how to add data sources for multiple rules. 1. Add a [rule](https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/observability_pipeline#nested-schema-for-configprocessor_groupprocessorsensitive_data_scanner) block in your Observability Pipeline resource for the library rule. ```terraform ... sensitive_data_scanner { rule { name = "" tags = [] on_match { redact { replace = "***" } } pattern { library { id = data.datadog_sensitive_data_scanner_standard_pattern..id use_recommended_keywords = true } } scope { all = true } } } ``` Replace the placeholders: - `` with a name for the rule. This name is shown in the Pipelines UI. - `` with the rule identifier you used in the data source in step 1. For example, if you use the [AWS Access Key ID Scanner](https://docs.datadoghq.com/security/sensitive_data_scanner/scanning_rules/library_rules/?search=AWS+Access+Key+ID+Scanner) data source from step 1, configure the rule block as follows: ```terraform ... sensitive_data_scanner { rule { name = "Redact AWS Access Key IDs" tags = [] on_match { redact { replace = "***" } } pattern { library { id = data.datadog_sensitive_data_scanner_standard_pattern.aws_access_key.id use_recommended_keywords = true } } scope { all = true } } } ``` See the full configuration example on how to add multiple rules. 1. Repeat steps 1 and 2 for all library rules you want to add. ### Full configuration example{% #full-configuration-example %} {% image source="https://datadog-docs.imgix.net/images/observability_pipelines/processors/sds_tf_ui.7727712c35cf1ff6544b402a8b815fb2.png?auto=format" alt="The Sensitive Data Scanner processor panel showing two scanning rules: Redact AWS Access Key IDs and Redact US SSNs" /%} If you want to use the Sensitive Data Scanner processor to scan for AWS Access Key IDs and US Social Security Numbers, and redact them by replacing them with the string `***`: 1. Use the [Datadog Sensitive Data Scanner Standard Pattern](https://registry.terraform.io/providers/DataDog/datadog/latest/docs/data-sources/sensitive_data_scanner_standard_pattern) data source to retrieve the rule IDs for the [AWS Access Key ID Scanner](https://docs.datadoghq.com/security/sensitive_data_scanner/scanning_rules/library_rules/?search=AWS+Access+Key+ID+Scanner) and the [US Social Security Number Scanner](https://docs.datadoghq.com/security/sensitive_data_scanner/scanning_rules/library_rules/?search=US+Social+Security+Number+Scanner). 1. In your [Datadog Observability Pipeline](https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/observability_pipeline) resource's Sensitive Data Scanner processor, use the Sensitive Data Scanner rules defined in the data sources. ```terraform data "datadog_sensitive_data_scanner_standard_pattern" "aws_access_key" { filter = "AWS Access Key ID Scanner" } data "datadog_sensitive_data_scanner_standard_pattern" "us_ssn" { filter = "US Social Security Number Scanner" } resource "datadog_observability_pipeline" "sensitive_data_pipeline" { name = "Sensitive Data Pipeline" config { source { id = "source-0" datadog_agent {} } processor_group { display_name = "Processors" enabled = true id = "group-0" include = "*" inputs = ["source-0"] processor { display_name = "Sensitive Data Scanner" enabled = true id = "processor-sds-0" include = "*" sensitive_data_scanner { rule { name = "Redact AWS Access Key IDs" tags = [] on_match { redact { replace = "***" } } pattern { library { id = data.datadog_sensitive_data_scanner_standard_pattern.aws_access_key.id use_recommended_keywords = true } } scope { all = true } } rule { name = "Redact US SSNs" tags = [] on_match { redact { replace = "***" } } pattern { library { id = data.datadog_sensitive_data_scanner_standard_pattern.us_ssn.id use_recommended_keywords = true } } scope { all = true } } } } } destination { id = "destination-0" inputs = ["group-0"] datadog_logs {} } } } ``` ## Further reading{% #further-reading %} - [Writing Effective Grok Parsing Rules with Regular Expressions](https://docs.datadoghq.com/logs/guide/regex_log_parsing/)