Jobs Monitoring for Azure Data Factory

Docs > Data Observability Overview > Data Observability: Jobs Monitoring > Jobs Monitoring for Azure Data Factory

Jobs Monitoring for Azure Data Factory is in Preview.

Overview

Data Observability: Jobs Monitoring gives visibility into the performance and reliability of your Azure Data Factory pipelines. Datadog reads pipeline and activity run history from the Data Factory management API. Each pipeline run appears as a job run with its activities broken out, along with the error message for failed runs.

Datadog also derives dataset lineage from the activities in your pipelines, connecting each pipeline to the tables and files it reads and writes.

Prerequisites

Before you begin, make sure you have:

An Azure subscription containing the data factories you want to monitor.
The Datadog Azure integration installed, with an App Registration that has access to that subscription.
Permission to create role assignments on the subscription or on individual data factories.

Add required Azure permissions

The Datadog Azure integration assigns the Monitoring Reader role. That role covers reading data factory metadata, but not reading run history. Azure registers the run-history endpoints as actions rather than reads, and the wildcard in Monitoring Reader matches reads only.

Grant the following three actions to the App Registration used by your Datadog Azure integration:

Microsoft.DataFactory/factories/querypipelineruns/action
Microsoft.DataFactory/factories/pipelineruns/queryactivityruns/action
Microsoft.DataFactory/factories/querytriggerruns/action

Without them, Datadog can list your factories and pipelines but collects no runs.

A custom role grants only the permissions Datadog uses. Save the following definition as datadog-adf-reader.json, replacing <SUBSCRIPTION_ID> with your subscription ID:

{
  "Name": "Datadog Data Observability - ADF Reader",
  "Description": "Read-only access to Azure Data Factory metadata and run history for Datadog Data Observability.",
  "IsCustom": true,
  "Actions": [
    "Microsoft.DataFactory/factories/read",
    "Microsoft.DataFactory/factories/*/read",
    "Microsoft.DataFactory/factories/querypipelineruns/action",
    "Microsoft.DataFactory/factories/querytriggerruns/action",
    "Microsoft.DataFactory/factories/pipelineruns/queryactivityruns/action"
  ],
  "NotActions": [],
  "DataActions": [],
  "NotDataActions": [],
  "AssignableScopes": ["/subscriptions/<SUBSCRIPTION_ID>"]
}

Create the role, then assign it to the App Registration:

az role definition create --role-definition ./datadog-adf-reader.json

az role assignment create \
  --assignee <APP_REGISTRATION_CLIENT_ID> \
  --role "Datadog Data Observability - ADF Reader" \
  --scope /subscriptions/<SUBSCRIPTION_ID>

To limit access to specific factories, repeat the assignment with a factory scope instead of the subscription scope:

--scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.DataFactory/factories/<FACTORY_NAME>

To assign the role across several subscriptions, set AssignableScopes to a management group, such as /providers/Microsoft.Management/managementGroups/<MANAGEMENT_GROUP_ID>.

Note: Creating a role definition requires the Owner or User Access Administrator role at the assignable scope.

If your organization does not allow custom roles, assign the built-in Data Factory Contributor role, which includes the three actions:

az role assignment create \
  --assignee <APP_REGISTRATION_CLIENT_ID> \
  --role "Data Factory Contributor" \
  --scope /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.DataFactory/factories/<FACTORY_NAME>

You can also assign it in the Azure portal under Data Factory > Access control (IAM) > Add role assignment.

This role also grants write and delete access on factories, pipelines, datasets, and linked services. Datadog uses only the read and query permissions. Scope the assignment to individual factories rather than to the whole subscription to limit what the role covers.

Configure the integration

Navigate to Datadog Data Observability > Settings > Integrations.
Click Configure next to Azure Data Factory.
Select the App Registration that has access to the subscriptions hosting your data factories.
Select the subscriptions to monitor, then click Save.

Datadog monitors every data factory in the subscriptions you select.

What Datadog collects

After setup, Datadog collects new pipeline and activity runs every few minutes. Runs from a pipeline you create after setup are collected on the same cadence.

For each pipeline run, Datadog reports:

The pipeline run and each of its activity runs, with start time, end time, and duration.
The run status, and the error message for failed runs.
The trigger that started the run.
An env tag set to the name of the data factory.
The annotations on the pipeline and the user properties on each activity, as tags. Tags you set in Data Factory are available for filtering and grouping in Datadog.

Dataset lineage

Datadog derives dataset lineage from the activities in each pipeline run, using the runtime values of the datasets and linked services the activity resolved. Because lineage comes from the run rather than from the pipeline definition, parameterized datasets resolve to the assets each run touched.

For activities that move data, such as a copy, Datadog reports the source and sink datasets. For activities that run SQL, Datadog reports the tables the statement read and wrote.

Datadog resolves datasets across the major warehouse, database, and object storage connectors, including:

Snowflake
Google BigQuery
Amazon Redshift
SQL Server and Azure SQL
Azure Databricks Delta Lake
Amazon S3
Azure Blob Storage
Azure Data Lake Storage

Assets resolve to the same entities that Datadog’s other Data Observability integrations report, so lineage connects across your data platforms.

Activities that Datadog cannot resolve datasets for appear as job runs without dataset lineage.

To see the lineage for a pipeline, open Lineage and anchor on the pipeline. The graph shows the tables the pipeline read upstream and the tables it wrote downstream.

The Lineage graph anchored on an Azure Data Factory pipeline, with the table it reads on the left and the three tables it writes on the right

Validate

Open Data Observability: Jobs Monitoring and set the job type selector to ADF Pipelines. Your pipelines appear with their run counts, failure rates, and durations, tagged with the name of their data factory.

Data Observability Jobs Monitoring overview filtered to ADF Pipelines, listing pipelines with their monitors, last status, duration, and failure rate

Click a pipeline to see how it performed across runs, then click a run to open it. The flame graph shows the pipeline run with each of its activities nested underneath, so you can see which activity took the longest. The Overview tab lists the run’s metadata, including the trigger that started it, and for failed runs the error message.

Flame graph for a single Azure Data Factory pipeline run, showing the pipeline span with its activity spans nested underneath, and the span Overview tab listing the trigger that started the run

Troubleshooting

No runs appear, but pipelines are listed

Datadog can read your factory metadata but not its run history. Confirm that the App Registration has the three action permissions described in Add required Azure permissions. Role assignments can take a few minutes to take effect.

Nothing appears at all

Confirm that the subscriptions hosting your data factories are selected in the integration configuration, and that the App Registration you selected has access to them. The Data Observability Settings page reports connection errors and warnings on the account.

Pipelines appear without dataset lineage

Not every activity carries the information Datadog needs to resolve the datasets it read and wrote. See Dataset lineage for what Datadog resolves. Pipelines built from activities it cannot resolve appear as job runs without lineage edges.