Quality Gates allows you to gate your workflows based on signals in Datadog. You can create two types of rules: Static Analysis and Tests.
For example, you can block code from being merged if it introduces new flaky tests or code security violations; issues that wouldn’t normally fail your CI/CD pipelines and end up deployed to production.
Using Quality Gates, you have control over what is merged into the default branch, and ultimately on what is deployed.
This allows you to ensure that the code running in production is meeting high quality standards, reducing incidents, and minimizing unwanted behaviors.
Set up Quality Gates
There are two main steps required to set up and use Quality Gates:
Create one or more rules on the Datadog site.
Invoke Quality Gates by using the datadog-ci gate evaluate command in your CI pipeline.
Create a rule
To create Quality Gates rules for your organization, your user account must have the quality_gate_rules_writepermission.
Define the rule scope. The rule scope defines when the rule should be evaluated. For example, you can create a rule that it is evaluated only on specific repositories and branches. To define the scope for a rule, switch to select when to evaluate and add included or excluded values for the scope. See rule scopes for more information.
Select the rule type. You can choose between Static Analysis and Test.
Define the rule condition. The rule condition states in which scenario the rule fails, failing the related pipeline (if the rule is blocking). You can select one of the existing rule conditions for the rule type you have selected.
The following example shows how to create a static analysis rule that fails when there are one or more static analysis violations with “error” status and “security” category being introduced in a specific commit:
Select whether the rule should block the pipeline or not when it fails.
Non-blocking rules are helpful when rolling out a new rule and wanting to verify its behavior before making it blocking.
Select the time window over which the query is evaluated.
Select a rule name that describes the rule that you are creating.
Click Save Rule.
Invoking quality gates
To use quality gates, your datadog-ci version should be 2.19.0 or later.
Retrieves all the rules that have rule scopes and custom scopes matching the current pipeline context (repository, branch, or custom scope passed in the command).
Evaluates all the matching rules.
If one or more blocking rules fail, the command fails as well, blocking the pipeline.
For the command to work properly, it's important that the events (tests, static analysis violations)
are sent to Datadog before the datadog-ci gate evaluate command is executed.
Otherwise, the rules might have an incorrect behavior due to the absence of the events.
The command requires the DD_API_KEY and DD_APP_KEY environment variables to point to your Datadog API Key
and Datadog Application Key. Also, you need to set the DD_BETA_COMMANDS_ENABLED environment
variable as true. Optionally, you can specify the DD_SITE environment variable to point to a specific datadog site (default value is datadoghq.com).
The behavior of the command can be modified using the following flags:
--fail-on-empty: when this flag is specified, the command fails if no matching rules were found in Datadog
based on the current pipeline context. By default, the command succeeds.
--fail-if-unavailable: when this flag is specified, the command fails if one or more rules could not be evaluated because of an internal issue.
By default, the command succeeds.
--timeout: the command stops its execution after the specified timeout in seconds. The default timeout is 10 minutes.
The command typically completes within a few minutes, but it could take longer due to processing delays in Datadog.
--no-wait: by default, the command waits a certain amount of time for the events (tests, static analysis violations) to arrive to Datadog.
This step is important as it makes sure that the events are queryable in Datadog before the rules are executed,
avoiding incorrect evaluations. If, in your pipeline, the job containing the datadog-ci gate evaluate command is
called several minutes after the related events are sent to Datadog, you could skip this waiting time by specifying the --no-wait flag.
If used incorrectly, this flag may result in inaccurate rule evaluations.
Custom scopes can be added by using the following option one or more times: --scope:
The command logs can be examined to understand what was the overall gate evaluation status, along with information
about all the rules that were evaluated.
Rule scope
When creating a rule, you can define a scope, which states when the rule should be evaluated. If the rule scope is
set to always evaluate, the rule is evaluated on all repositories and branches.
When the datadog-ci gate evaluate command is invoked, the rules having a scope matching the command context are evaluated.
For each scope (for example, branch), you can either select included or excluded values.
When included values are selected, the rule is evaluated if one or more included values are part of the command context.
When excluded values are selected, the rule is not evaluated if any of the excluded values are part of the command context.
For example, to create a rule that is evaluated in all branches except main of the example-repository repository, you can define the following scope:
If a rule does not contain a scope, it is evaluated for all values for that scope.
For example, if a rule does not contain the repository scope, it is evaluated for all repositories.
Custom scope
In addition to branch and repository, you can define custom scopes to filter rules that are evaluated for a specific CI pipeline.
To add a custom scope when creating a rule:
Click + Add Filter and select Custom Scope.
Define the scope name, for example, team.
Define the scope’s included or excluded values.
Unlike the branch and repository scopes, custom scopes need to be provided in the datadog-ci gate evaluate command using the --scope option.
For example, you can create a rule that is evaluated for the example-repository repository, but only when the team is backend:
The rule would be evaluated when this command is invoked in a CI pipeline of the example-repository repository:
datadog-ci gate evaluate --scope team:backend
The rule would not be evaluated when the following commands are invoked instead:
datadog-ci gate evaluate, as it does not define any team.
datadog-ci gate evaluate --scope team:api --scope team:frontend, as it defines teams different from backend.
Manage rules
To edit a Quality Gates rule, click the Edit icon to the right of the Creator avatar on the Quality Gates Rules page.
To delete a Quality Gates rule, click the Delete icon next to the Edit button on the Quality Gates Rules page. Alternatively, you can edit the rule and click Delete Rule.
Enable GitHub check creation
You can automatically create a GitHub check for each rule evaluated. When this feature is enabled, the evaluation results appear directly in GitHub.
The check contains additional information about the rule evaluation, such as the failure reason and the matching events in Datadog.
Grant Checks: Write access to the GitHub application.
After the permission is granted, you should be able to see the checks in GitHub.
Note: a current limitation is that you cannot re-run a check to re-run the related Quality Gates rule.
Permissions
Only users with the quality_gate_rules_write permission can create and edit Quality Gate rules. Users with the quality_gate_rules_read permission can view Quality Gate rules.
For more information, see the RBAC Permissions documentation.
Track changes in rules
You can view information about who created, modified, and deleted Quality Gates rules in Audit Trail.
Further Reading
Additional helpful documentation, links, and articles:
