Overview

You can use Terraform to manage your Incident Management configuration through the Datadog API. This guide covers the Incident Management resources available in the Terraform registry and links to the corresponding Datadog documentation for each.

Configuring incident types, fields, roles, and notification rules by hand in the UI works well for a handful of teams. It becomes harder to scale as your organization grows, for example, when standardizing configuration across hundreds of teams or migrating from another incident management tool. Terraform lets you define this configuration as code, so you can create and update it programmatically and keep it consistent across your organization.

You can also import your existing incident type, notification, and postmortem template configurations into Terraform, and reference existing configurations as Terraform data sources.

What you can manage with Terraform

Set up the Datadog Terraform provider

If you haven’t already, configure the Datadog Terraform provider to interact with Datadog APIs through a Terraform configuration.

Incident types

Incident types let you apply different settings, fields, roles, and notification behavior to different classes of incidents, such as security incidents versus customer-impacting incidents. Every other resource on this page is scoped to an incident type, so define your incident types first. Use the configuration block on the incident type resource to create incident types and set the toggles found on the Information page, such as incident deletion, test incidents, and private incidents. Severity and status levels, also found on the Information page, are configured with the incident user-defined field resource.

To learn how this works in Datadog, see Incident Types.

Property fields

Property fields let responders capture structured data on an incident, for example, root cause or affected region. Use the incident user-defined field resource to create property fields and scope them to an incident type.

To learn how this works in Datadog, see Property Fields.

Responder types

Responder types define the roles people can be assigned during an incident. Examples include Incident Commander or a custom role like Comms Lead. Use the incident user-defined role resource to create custom responder types. Scope each one to an incident type.

To learn how this works in Datadog, see Responder Types.

Notification rules

Notification rules determine when a notification fires, who it’s sent to, and which template it uses. Use the incident notification rule resource to create rules based on triggers such as incident creation or a saved change. Conditions on a rule can include severity or affected services.

To learn how this works in Datadog, see Notification Rules.

Postmortem templates

Postmortem templates control where a postmortem document is generated for an incident type. Templates standardize the content a postmortem writer is expected to populate by defining specific sections and headers in the document. Use the incident postmortem template resource to configure this per incident type.

To learn how this works in Datadog, see Postmortem Templates.

Notification templates

Notification templates define reusable message content for incident notifications. Use the incident notification template resource to create templates scoped to an incident type.

To learn how this works in Datadog, see Notification Templates.

Full configuration example

The following example combines several of these resources into one configuration:

• A datadog_incident_type with a configuration block that disables incident deletion and enables test incidents
• A datadog_incident_user_defined_field and datadog_incident_user_defined_role, both scoped to that type
• A datadog_incident_notification_template
• A datadog_incident_notification_rule that uses the template

resource "datadog_incident_type" "customer_impacting" {
  name        = "Customer Impacting"
  description = "Incidents that impact customers"
  configuration = {
    private_incidents            = false
    private_incidents_by_default = false
    allow_workflows              = true
    allow_incident_deletion      = false
    editable_timestamps          = false
    test_incidents               = true
    create_message               = ""
    slug_source                  = "default"
  }
}

resource "datadog_incident_user_defined_field" "root_cause" {
  name          = "root_cause"
  type          = "dropdown"
  incident_type = datadog_incident_type.customer_impacting.id

  valid_value {
    display_name = "Service Bug"
    value        = "service_bug"
  }
}

resource "datadog_incident_user_defined_role" "tech_lead" {
  name          = "Tech Lead"
  incident_type = datadog_incident_type.customer_impacting.id
}

resource "datadog_incident_notification_template" "sev1_alert" {
  name          = "SEV-1 Customer Impact Template"
  subject       = "SEV-1 Incident: {{incident.title}}"
  category      = "alert"
  incident_type = datadog_incident_type.customer_impacting.id
  content       = "SEV-1 declared: {{incident.title}}. Status: {{incident.status}}."
}

resource "datadog_incident_notification_rule" "sev1_sev2_created" {
  enabled               = true
  trigger               = "incident_created_trigger"
  visibility            = "organization"
  handles               = ["@pagerduty-on-call"]
  incident_type         = datadog_incident_type.customer_impacting.id
  notification_template = datadog_incident_notification_template.sev1_alert.id

  conditions {
    field  = "severity"
    values = ["SEV-1", "SEV-2"]
  }
}

Further reading

Additional helpful documentation, links, and articles:

Learn more about Incident ManagementDOCUMENTATION Using test incidents for training and testingDOCUMENTATION Terraform provider for DatadogDOCUMENTATION