---
title: dbt
description: >-
  Connect dbt Cloud or dbt Core to Datadog for job run metadata and model
  lineage.
breadcrumbs: 'Docs > Data Observability Overview > Data Observability: Jobs Monitoring > dbt'
---

# dbt

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ().
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Datadog can access your dbt Cloud or dbt Core metadata to extract information about job runs, including run durations, models generated by dbt, and lineage relationships between models. Datadog matches tables in your warehouse with dbt models to determine the causality and consequences of table failure.

{% tab title="dbt Cloud" %}
Follow the steps below to connect dbt Cloud to Datadog.

## Generate an API token in dbt Cloud{% #generate-an-api-token-in-dbt-cloud %}

Create a service token in dbt Cloud so Datadog can access your account's metadata.

1. In dbt Cloud, go to User Profile > API Tokens > Service Tokens.
1. Click on + Create Service Token.
1. Provide a name for the token.
1. Set the token permissions:
   - If you create the webhook in dbt Cloud yourself, use the Stakeholder/Read-Only permission set scoped to the relevant dbt Cloud projects.
   - If Datadog creates and manages the webhook, use Developer permissions for the dbt Cloud Enterprise plan or Account Admin permissions for the dbt Cloud Team plan.
1. Click Save and copy the generated API token.

## Connect your dbt Cloud account to Datadog{% #connect-your-dbt-cloud-account-to-datadog %}

Use the API token to configure the integration in Data Observability.

1. Navigate to [Datadog Data Observability > Settings](https://app.datadoghq.com/data-obs/settings/integrations).
1. In the dbt Cloud section, click Configure.
1. If you have already created a dbt Cloud integration account, make sure you have updated it with the API token with permissions described above.
1. If not, create an account. Fill in the Account Name, Account Id, Account Url, and API Token sections.
1. Click Save to save your settings.

## Configure webhooks{% #configure-webhooks %}

In Data Observability settings, expand the dbt Cloud account and choose how Datadog receives dbt Cloud job run events.

### Create the webhook in dbt Cloud yourself{% #create-the-webhook-in-dbt-cloud-yourself %}

Use this option if you want to use a Stakeholder/Read-Only service token for artifact ingestion.

1. Select I'll manage the webhook in dbt Cloud myself.
1. Copy the Datadog webhook URL.
1. In dbt Cloud, go to Account Settings > Webhooks > Create New Webhook.
1. Paste the Datadog webhook URL into the webhook URL field.
1. Enable the Job Run Started and Job Run Completed events. To scope ingestion to specific jobs, select those jobs in the dbt Cloud webhook configuration.
1. Save the webhook in dbt Cloud.
1. Copy the HMAC secret from dbt Cloud, paste it into the HMAC secret from dbt Cloud field in Datadog, and click Save.

**Note**: After saving, webhooks you create yourself can take up to 5 minutes to start accepting traffic from dbt Cloud.

If you remove a user-managed webhook configuration in Datadog later, delete the webhook from dbt Cloud manually.

### Let Datadog manage the webhook{% #let-datadog-manage-the-webhook %}

Use this option if you want Datadog to create and maintain the webhook in dbt Cloud.

1. Select Datadog-managed.
1. Click Save.

This mode requires a dbt Cloud token with Developer permissions for the dbt Cloud Enterprise plan or Account Admin permissions for the dbt Cloud Team plan.

## What's next{% #whats-next %}

After your next dbt job run, you should start seeing job run and lineage data in [Datadog Data Observability](https://app.datadoghq.com/data-obs/catalog?integration=dbt), as shown below.

{% image
   source="https://docs.dd-static.net/images/data_observability/data-obs-dbt-cloud-final.0a9482f0c463f82642a440c622b35e28.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/data_observability/data-obs-dbt-cloud-final.0a9482f0c463f82642a440c622b35e28.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Data Observability overview showing dbt job runs as a stacked bar chart over time and a table of connected dbt Cloud accounts with their status." /%}

{% /tab %}

{% tab title="dbt Core" %}
Follow the steps below to connect dbt Core to Datadog.

**Note**: If you run dbt Core with an external orchestrator (such as Airflow) and want to correlate orchestrator tasks with dbt runs, follow the [Airflow integration instructions](https://docs.datadoghq.com/data_jobs/airflow.md?tab=kubernetes) first.

## Retrieve your Datadog API key{% #retrieve-your-datadog-api-key %}

1. [Follow these instructions](https://docs.datadoghq.com/account_management/api-app-keys.md#add-an-api-key-or-client-token) to create or retrieve a Datadog API key.

## Install openlineage-dbt{% #install-openlineage-dbt %}

1. Install the `openlineage-dbt` package. Reference [Using dbt with Amazon MWAA](https://docs.aws.amazon.com/mwaa/latest/userguide/samples-dbt.html) for setting up this package in your virtual environment.

   ```shell
   pip3 install openlineage-dbt>=1.39.0
   ```

## Set the environment variables{% #set-the-environment-variables %}

1. Set the following environment variables. Replace `datadoghq.com` with the relevant [Datadog site](https://docs.datadoghq.com/getting_started/site.md#access-the-datadog-site) for your organization. For more information on predefined Datadog sites, see the [OpenLineage documentation](https://openlineage.io/docs/client/python/#predefined-datadog-sites).

   ```shell
   export DD_SITE=datadoghq.com
   export DD_API_KEY=<YOUR_DATADOG_API_KEY>
   export OPENLINEAGE__TRANSPORT__TYPE=datadog
   
   # OPENLINEAGE_NAMESPACE determines the Datadog tag value for the environment (similar to how the service tag identifies the application).
   # Typical values are dev, staging, or prod, but you can over ride it with any custom value.
   export OPENLINEAGE_NAMESPACE=<YOUR_ENV>
   
   # Optional, for debugging purposes
   export OPENLINEAGE_CLIENT_LOGGING=DEBUG
   ```

## Update the dbt invocation{% #update-the-dbt-invocation %}

1. Change your dbt invocations to use the OpenLineage wrapper (`dbt-ol`) instead of calling `dbt` directly. This applies to any dbt command you want to track in Datadog, such as `run`, `build`, and `test`. For the full list of available commands, see the [dbt documentation](https://docs.getdbt.com/docs/running-a-dbt-project/run-your-dbt-projects).

1. Add the `--consume-structured-logs` flag to view dbt jobs while the command is still running.

   ```shell
   # Run models
   dbt-ol run --consume-structured-logs --openlineage-dbt-job-name <YOUR_DBT_JOB_NAME>
   
   # Run tests (required to see test failures in Datadog)
   dbt-ol test --consume-structured-logs --openlineage-dbt-job-name <YOUR_DBT_JOB_NAME>
   
   # Run build (runs models, tests, seeds, and snapshots)
   dbt-ol build --consume-structured-logs --openlineage-dbt-job-name <YOUR_DBT_JOB_NAME>
   ```

## What's next{% #whats-next %}

After your next dbt job run, you should start seeing job run and lineage data in [Datadog Data Observability](https://app.datadoghq.com/data-obs/catalog?integration=dbt), as shown below.

{% image
   source="https://docs.dd-static.net/images/data_observability/data-obs-dbt-cloud-final.0a9482f0c463f82642a440c622b35e28.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/data_observability/data-obs-dbt-cloud-final.0a9482f0c463f82642a440c622b35e28.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Data Observability overview showing dbt job runs and model lineage." /%}

{% /tab %}

## Further reading{% #further-reading %}

- [Learn about Data Observability](https://docs.datadoghq.com/data_observability.md)
- [Understanding dbt: basics and best practices](https://www.datadoghq.com/blog/understanding-dbt/)