dbt

dbt integrations are in Preview.

Overview

Datadog can access your dbt Cloud or dbt Core metadata to extract information about job runs, including run durations, models generated by dbt, and lineage relationships between models. Datadog matches tables in your warehouse with dbt models to determine the causality and consequences of table failure.

    Follow the steps below to connect dbt Cloud to Datadog.

    Generate an API token in dbt Cloud

    Create a service token in dbt Cloud so Datadog can access your account’s metadata.

    1. In dbt Cloud, go to User Profile > API Tokens > Service Tokens.
    2. Click on + Create Service Token.
    3. Provide a name for the token.
    4. Set the token permissions:
      • For dbt Cloud Enterprise plan, ensure that the token has Developer permissions.
      • For dbt Cloud Team plan, ensure that the token has Account Admin permissions.
    5. Click Save and copy the generated API token.

    Connect your dbt Cloud account to Datadog

    Use the API token to configure the integration in Datadog.

    1. Navigate to Datadog’s dbt Cloud integration tile.
    2. If you have already created a dbt Cloud integration account, make sure you have updated it with the API token with permissions described above.
    3. If not, create a new account. Fill in the Account Name, Account Id, Account Url, and API Token sections.
    4. Click Save to save your settings.

    What’s next

    After your next dbt job run, you should start seeing job run and lineage data in Datadog Data Observability, as shown below.

    Data Observability overview showing dbt job runs as a stacked bar chart over time and a table of connected dbt Cloud accounts with their status.

    Follow the steps below to connect dbt Core to Datadog.

    Note: If you run dbt Core with an external orchestrator (such as Airflow) and want to correlate orchestrator tasks with dbt runs, follow the Airflow integration instructions first.

    Retrieve your Datadog API key

    1. Follow these instructions to create or retrieve a Datadog API key.

    Install openlineage-dbt

    1. Install the openlineage-dbt package. Reference Using dbt with Amazon MWAA for setting up this package in your virtual environment.

      pip3 install openlineage-dbt>=1.39.0

    Set the environment variables

    1. Set the following environment variables. Replace datadoghq.com with the relevant Datadog site for your organization. For more information on predefined Datadog sites, see the OpenLineage documentation.

      export DD_SITE=datadoghq.com
export DD_API_KEY=<YOUR_DATADOG_API_KEY>
export OPENLINEAGE__TRANSPORT__TYPE=datadog

# OPENLINEAGE_NAMESPACE determines the Datadog tag value for the environment (similar to how the service tag identifies the application).
# Typical values are dev, staging, or prod, but you can over ride it with any custom value.
export OPENLINEAGE_NAMESPACE=<YOUR_ENV>

# Optional, for debugging purposes
export OPENLINEAGE_CLIENT_LOGGING=DEBUG

    Update the dbt invocation

    1. Change the dbt invocation to use the OpenLineage wrapper (dbt-ol).

    2. Add the --consume-structured-logs flag to view dbt jobs while the command is still running.

      dbt-ol run --consume-structured-logs --openlineage-dbt-job-name <YOUR_DBT_JOB_NAME>

    What’s next

    After your next dbt job run, you should start seeing job run and lineage data in Datadog Data Observability, as shown below.

    Data Observability overview showing dbt job runs and model lineage.

    Further reading

    Additional helpful documentation, links, and articles:

    Learn about Data ObservabilityDOCUMENTATION more more Understanding dbt: basics and best practicesBLOG more more