Overview

Sensitive data, such as credit card numbers, bank routing numbers, and API keys are often exposed unintentionally in application logs, APM spans, and RUM events, which can expose your organization to financial and privacy risks.

Sensitive Data Scanner is a stream-based, pattern matching service used to identify, tag, and optionally redact or hash sensitive data. Security and compliance teams can implement Sensitive Data Scanner as a new line of defense, helping prevent against sensitive data leaks and limiting non-compliance risks.

To use Sensitive Data Scanner, set up a scanning group to define what data to scan and then set up scanning rules to determine what sensitive information to match within the data.

If you want to redact your sensitive data in your environment before shipping to your downstream destinations, see how to redact sensitive data with Observability Pipelines.

This document walks you through the following:

• The permissions required to view and set up Sensitive Data Scanner.
• Setting up scanning for sensitive data.
• Using the out-of-the-box dashboard.

Note: See PCI DSS Compliance for information on setting up a PCI-compliant Datadog organization.

Permissions

By default, users with the Datadog Admin role have access to view and set up scanning rules. To allow other users access, grant the data_scanner_read or data_scanner_write permissions under Compliance to a custom role. See Access Control for details on how to set up roles and permissions.

Set up Sensitive Data Scanner

This section walks you through the following to set up Sensitive Data Scanner:

• Adding a scanning group
• Adding scanning rules
• Controlling access to events with sensitive data

You can also redact sensitive data in tags.

Add a scanning group

A scanning group determines what data to scan. It consists of a query filter and a set of toggles to enable scanning for logs, APM, RUM, and events. See the Log Search Syntax documentation to learn more about query filters.

For Terraform, see the Datadog Sensitive Data Scanner group resource.

To set up a scanning group:

1. Navigate to the Sensitive Data Scanner configuration page. Alternatively, go to Organization Settings > Manage Sensitive Data and click the Configuration tab.
2. Click Add scanning group. Alternatively, click the Add dropdown menu on the top right corner of the page and select Add Scanning Group.
3. Enter a query filter for the data you want to scan. At the top, click APM Spans to preview the filtered spans. Click Logs to see the filtered logs.
4. Enter a name and description for the group.
5. Click the toggle buttons to enable Sensitive Data Scanner for the products you want (for example, logs, APM spans, RUM events, and Datadog events).
6. Click Create.

Add scanning rules

A scanning rule determines what sensitive information to match within the data defined by a scanning group. You can add predefined scanning rules from Datadog’s Scanning Rule Library or create your own rules using regex patterns. The data is scanned at ingestion time during processing. For logs, this means the scan is done before indexing and other routing decisions.

For Terraform, see the Datadog Sensitive Data Scanner rule resource.

To add scanning rules:

1. Navigate to the Sensitive Data Scanner configuration page.
2. Click the scanning group where you want to add the scanning rules.
3. Click Add Scanning Rule. Alternatively, click the Add dropdown menu on the top right corner of the page and select Add Scanning Rule.
4. Select whether you want to add a library rule or create a custom scanning rule.

Notes:

• Any rules that you add or update only affect data coming into Datadog after the rule was defined.
• Sensitive Data Scanner does not affect any rules you define on the Datadog Agent directly.
• To turn off Sensitive Data Scanner entirely, set the toggle to off for each Scanning Group and Scanning Rule so that they are disabled.

See Investigate Sensitive Data Issues for details on how to use the Summary page to triage your sensitive data issues.

Control access to logs with sensitive data

To control who can access logs containing sensitive data, use tags added by the Sensitive Data Scanner to build queries with role-based access control (RBAC). You can restrict access to specific individuals or teams until the data ages out after the retention period. See How to Set Up RBAC for Logs for more information.

Redact sensitive data in tags

To redact sensitive data contained in tags, you must remap the tag to an attribute and then redact the attribute. Uncheck Preserve source attribute in the remapper processor so that the tag is not preserved during the remapping.

To remap the tag to an attribute:

1. Navigate to your log pipeline.
2. Click Add Processor.
3. Select Remapper in the processor type dropdown menu.
4. Name the processor.
5. Select Tag key(s).
6. Enter the tag key.
7. Enter a name for the attribute the tag key is remapped to.
8. Disable Preserve source attribute.
9. Click Create.

To redact the attribute:

1. Navigate to your scanning group.
2. Click Add Scanning Rule.
3. Check the library rules you want to use.
4. Select Specific Attributes for Scan entire event or portion of it.
5. Enter the name of the attribute you created earlier to specify that you want it scanned.
6. Select the action you want when there’s a match.
7. Optionally, add tags.
8. Click Add Rules.

Out-of-the-box dashboard

When Sensitive Data Scanner is enabled, an out-of-the-box dashboard summarizing sensitive data findings is automatically installed in your account. To access this dashboard, go to Dashboards > Dashboards List and search for Sensitive Data Scanner Overview.

Further Reading

Additional helpful documentation, links, and articles:

Reducing data related risksDOCUMENTATION Regular expression syntax for custom scanning rulesDOCUMENTATION Discover, triage, and remediate sensitive data issues at scale with Sensitive Data ScannerBLOG Build a modern data compliance strategy with Datadog's Sensitive Data ScannerBLOG Best practices for sensitive data managementBLOG