Create your DDR org

1. Go to Get Started with Datadog. You may need to log out of your current session, or use incognito mode, to access this page.
2. Choose a different Datadog site than your primary (for example, if you’re on US1, choose EU or US5). See the Datadog site list for options. All Datadog sites are geographically separated.
3. Follow the prompts to create an account.

Share the DDR org information with Datadog

Email your new org name to your Customer Success Manager. Then, your Customer Success Manager sets this new org as your DDR org.

After Datadog designates your DDR organization, link your DDR org to your primary org:

1. Use the List your managed organizations endpoint to retrieve the public IDs of your primary and DDR organizations.

2. Add the disaster_recovery_status_write scope to your application key in the primary org.

3. Run the following commands, replacing the placeholders with the appropriate values.

   export PRIMARY_DD_API_KEY=<PRIMARY_ORG_API_KEY>
   export PRIMARY_DD_APP_KEY=<PRIMARY_ORG_APP_KEY>
   export PRIMARY_DD_API_URL=<PRIMARY_ORG_API_SITE>
   
   export DDR_ORG_ID=<DDR_ORG_PUBLIC_ID>
   export PRIMARY_ORG_ID=<PRIMARY_ORG_PUBLIC_ID>
   export USER_EMAIL=<USER_EMAIL>
   export CONNECTION='{"data":{"id":"'${PRIMARY_ORG_ID}'","type":"hamr_org_connections","attributes":{"TargetOrgUuid":"'${DDR_ORG_ID}'","HamrStatus":1,"ModifiedBy":"'${USER_EMAIL}'", "IsPrimary":true}}}'
   
   curl -v -H "Content-Type: application/json" -H \
   "dd-api-key:${PRIMARY_DD_API_KEY}" -H \
   "dd-application-key:${PRIMARY_DD_APP_KEY}" --data "${CONNECTION}" --request POST ${PRIMARY_DD_API_URL}/api/v2/hamr
   

After you link your orgs, the failover org displays this banner:

Datadog recommends using Single Sign On (SSO) to enable all your users to log in to your Disaster Recovery org during an outage.

Go to Organization Settings in your DDR org to configure SAML or Google Login for your users.

Managed sync replicates user accounts from your primary org to your DDR org. Datadog recommends configuring Just-in-Time provisioning with SAML so users can access the DDR org during a failover without needing to reset their password.

See the AWS, Azure, and Google Cloud integrations for setup steps.

Your cloud integrations must be configured in both primary and DDR orgs, but they only run in one org at a time: by default in the primary org, and during failover in the DDR org.

For more information, see the Cloud integrations failover section.

Datadog manages resource sync on your behalf using the open source datadog-sync-cli tool. You do not need to run or operate this tool yourself.

Managed sync replicates resources from your primary org to your DDR org on a regular schedule. Replicated resources include dashboards, monitors, users, notebooks, and 34+ other resource types. Replication runs on this schedule so your DDR org stays current before an outage.

Users are scoped to each Datadog site. Managed sync replicates user accounts to your DDR org. However, users may need to reset their password on first login to the DDR org. Datadog recommends configuring Just-in-Time provisioning with SAML so users can access the DDR org without manual password resets.

Managed sync uses a Datadog service account. During onboarding, create a service account in your DDR org to read and replicate resources from your primary org. Resources synced by managed sync are provisioned by a user mapped to their original owner when possible.

Remote Configuration (RC) allows you to remotely configure and change the behavior of Datadog Agents deployed in your infrastructure.

Remote Configuration is enabled by default for new orgs, including your DDR org. Any new API keys you create are RC-enabled for use with your Agent. For more details, see the Remote Configuration documentation.

Datadog strongly recommends using Remote Configuration for better failover control. As an alternative to RC, you can manually configure your Agents or use configuration management tools such as Puppet, Ansible, or Chef.

To enable Dual Shipping, Datadog recommends using Fleet Automation for management at scale. Alternatively, you can configure it manually by editing your datadog.yaml file.

Contact your Datadog Customer Success Manager to schedule dedicated time windows for failover testing to measure performance and Recovery Time Objective (RTO).

Go to Fleet Automation > Configure Agents in your DDR org to create a failover policy or reuse an existing one, and apply it to your fleet of Agents. Soon after the policy is enabled, Agents begin dual-shipping telemetry to both the primary and DDR (failover) observability sites.

To create a failover policy, click Create Failover Policy.

Then, follow the prompt to scope the hosts and telemetry (metrics, logs, traces) that you are required to fail over.

Cloud Integrations can only run in either your primary or DDR Datadog site, but not both at the same time, so failing them over ceases Cloud Integration data in your primary site. During an integration failover, integrations run only in the DDR data center. When no longer in failover, disable the failover policy to return integration data collection to the primary org.

During a failover or failover exercises, update your Datadog Agent’s datadog.yaml configuration file as shown in the following example, and restart the Agent.

• enabled: true allows the Agent to send metadataData about the Agent and the infrastructure host. For example, host name, host tags, Agent version. to the DDR Datadog site so you can view Agents and your Infra hosts in the DDR org. This allows you to see your Agents and infrastructure hosts in the failover org.

• failover_metrics, failover_logs, and failover_apm are false by default. Setting these to true causes the Agent to start sending telemetryData that is sent to the Datadog platform. For example, logs, metrics, traces. to the DDR org.

multi_region_failover:
  enabled: true
  failover_metrics: false
  failover_logs: false
  failover_apm: false
  site: <DDR_SITE>  # For example, "site: us5.datadoghq.com" for a US5 site
  api_key: <DDR_SITE_API_KEY>

DNS-based failover is a complementary approach to Agent-based failover. Instead of configuring Agents with a secondary site endpoint, you configure all your data sources to send telemetry to a single Datadog-provided custom intake URL. During a failover event, Datadog updates the DNS record for that URL to redirect traffic from your primary site to your DDR site.

Receive your custom DNS endpoint

If you choose to use DNS-based failover, Datadog provisions a custom intake URL for your organization (for example, <your-org>.intake.datadoghq.com). Configure all your data sources (such as Agents, log shippers, and custom instrumentation) to send telemetry to this endpoint instead of to the default Datadog intake URL. This is a one-time configuration change.

Trigger a DNS failover

Contact your Customer Success Manager or Datadog Support to initiate a DNS failover. Datadog updates the DNS record to redirect traffic from your primary site to your DDR site. The target Recovery Time Objective (RTO) is 2 hours from the time failover is initiated.

To trigger a failover of your Agents, click one of the policies in Fleet Automation in your DDR org, then click Enable. The status of each host updates as the failover occurs.

Use the steps appropriate for your environment to activate/test the DDR failover.

agent config set multi_region_failover.failover_metrics true
agent config set multi_region_failover.failover_logs true
agent config set multi_region_failover.failover_apm true

kubectl exec <POD_NAME> -c agent -- agent config set multi_region_failover.failover_metrics true
kubectl exec <POD_NAME> -c agent -- agent config set multi_region_failover.failover_logs true
kubectl exec <POD_NAME> -c agent -- agent config set multi_region_failover.failover_apm true

multi_region_failover:
  enabled: true
  failover_metrics: true
  failover_logs: true
  failover_apm: true
  site: NEW_ORG_SITE
  api_key: NEW_SITE_API_KEY

DD_MULTI_REGION_FAILOVER_ENABLED=true
DD_MULTI_REGION_FAILOVER_FAILOVER_METRICS=true
DD_MULTI_REGION_FAILOVER_FAILOVER_LOGS=true
DD_MULTI_REGION_FAILOVER_FAILOVER_APM=true
DD_MULTI_REGION_FAILOVER_SITE=ADD_NEW_ORG_SITE
DD_MULTI_REGION_FAILOVER_API_KEY=ADD_NEW_SITE_API_KEY

You can test failover for your cloud integrations from your DDR organization’s landing page.

On the failover landing page, you can check the status of your DDR org, or click Fail over your integrations to test your cloud integration failover.

When no longer in failover, disable the failover policy in the DDR org to return integration data collection to the primary org.

During testing, integration telemetry is spread over both organizations. If you cancel a failover test, the integrations return to running in the primary data center.