Overview

Datadog Adaptive Sampling helps you capture more relevant traces while remaining close to a specific budget (ingested gigabytes). To get started:

1. Set a target monthly volume for trace ingestion
2. Enroll one or more services to adaptive sampling. This ensures that the consumption of these services matches the target volume at the end of the month, while keeping visibility over its endpoints.

Adaptive Sampling relies on remote configuration and on the existing sampling rules mechanisms to dynamically adjust sampling rates for each environment, service and resource combination, in order to:

• match your specified monthly budget
• ensure visibility for low-traffic services and endpoints by capturing at least one trace for each combination of service, resource and environment every 5 minutes.

To enroll new services to adaptive sampling and manage ingested volumes from the Datadog Ingestion control page, follow the instructions listed below.

Requirements

• Datadog Agent 7.53.0 or higher.
• Remote Configuration enabled for your Agent.
• APM Remote Configuration Write permissions. If you don’t have these permissions, ask your Datadog admin to update your permissions from your organization settings.

Tracing library versions

The following table lists minimum tracing library versions required for Adaptive Sampling:

View sampling rates by resource

To see configured sampling rates:

1. Navigate to the Ingestion Controls Service Ingestion Summary.
2. View the table listing the applied sampling rates by resource of the service.

The table includes:

• Ingested bytes: Ingested bytes from spans of the service and resource.
• Downstream bytes: Ingested bytes from spans where the sampling decision starts from that service and resource, including downstream services.
• Configuration: Source of the resource sampling rate:
  • AUTOMATIC: Default head-based sampling mechanism from the Agent.
  • LOCAL CONFIGURED: Sampling rule set locally in the tracing library.
  • REMOTE CONFIGURED: Remote sampling rule set from the Datadog UI.
  • ADAPTIVE REMOTE: Adaptive sampling rules set by Datadog.

Configure adaptive sampling for a service

To configure adaptive sampling for the service:

1. Navigate to the Datadog Ingestion Control page.

2. Open the modal to set/edit the Monthly Ingestion Target for adaptive sampling. Make sure that the ingestion volume target is >0 when enrolling a first service to adaptive sampling. For subsequent services, you can increase the allocated budget after the new service is onboarded to account for the new volume.

Note: the configured budget is only allocated to services enrolled to adaptive sampling. It does not account for the additional volume from services that use local sampling rules or other sampling mechanisms configured locally in the Agent or the tracing library.

3. Navigate to the Service Ingestion Summary page for your service.
4. Click Manage Ingestion Rate. If the remote configuration option is disabled, make sure that the listed requirements are all met.
5. Set your service’s sampling strategy to Datadog Adaptive Sampling Rates and click Apply to save the configuration.
6. [Optional] Configure explicit sampling rates for specific resources, for which you would like to capture more (e.g. 100% of GET /checkout endpoints) or less (e.g. 0.1% of /health requests) data.

The configuration should take effect in 5-6 minutes, the time it takes for Datadog to observe the service’s traffic pattern, compute, then apply the sampling rates. Resources controlled by adaptive sampling will appear marked as Adaptive Remote in the Configuration column.

Further reading

Más enlaces, artículos y documentación útiles:

Ingestion MechanismsDOCUMENTATION Ingestion Control PageDOCUMENTATION