--- title: Upgrade OpenLineage Provider on Amazon MWAA for Airflow 2.7.2, 2.8.1, or 2.9.2 description: >- Resolve dependency conflicts when installing apache-airflow-providers-openlineage on Amazon MWAA Airflow 2.7.2, 2.8.1, or 2.9.2. breadcrumbs: >- Docs > Data Observability Overview > Data Observability: Jobs Monitoring > Upgrade OpenLineage Provider on Amazon MWAA for Airflow 2.7.2, 2.8.1, or 2.9.2 --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Upgrade OpenLineage Provider on Amazon MWAA for Airflow 2.7.2, 2.8.1, or 2.9.2 {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} Use this guide if your Amazon MWAA environment runs Airflow `2.7.2`, `2.8.1`, or `2.9.2`. For these Airflow versions, MWAA default constraints pin older `apache-airflow-providers-openlineage` and OpenLineage package versions. These versions can cause known reliability and compatibility issues, which can degrade the Data Observability experience. To use provider versions that include these fixes, update constraints and requirements together. Amazon MWAA enforces package constraints for each Airflow and Python version, so upgrading OpenLineage packages can conflict with MWAA defaults unless you provide custom constraints. For details about MWAA dependency and constraints behavior, see AWS documentation on [Python dependencies](https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html). To review provider compatibility and requirements, see [OpenLineage provider documentation](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html). For base setup steps, see [Enable Data Observability: Jobs Monitoring for Apache Airflow](https://docs.datadoghq.com/data_observability/jobs_monitoring/airflow.md?tab=amazonmwaa). ## For other MWAA versions{% #for-other-mwaa-versions %} If your MWAA environment runs an Airflow version that is not listed in this guide, Datadog recommends using the newest `apache-airflow-providers-openlineage` release that is compatible with your Airflow version. Use the [Airflow OpenLineage provider compatibility table](https://docs.datadoghq.com/data_observability/jobs_monitoring/airflow.md#openlineage-provider-compatibility) to choose that release, then follow the same constraints pattern: 1. Download the MWAA constraints file for your Airflow and Python versions. 1. Update the OpenLineage provider pin and any required OpenLineage dependency pins in `constraints.txt`. 1. Pin the same versions in `requirements.txt`. 1. Upload both files to your MWAA S3 bucket and update the MWAA environment. Datadog recommends using the newest compatible provider release available for your Airflow version. Review the [OpenLineage provider changelog](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/changelog.html) before updating, because provider releases can change the minimum supported Airflow version or required dependency versions. ## Requirements{% #requirements %} - Access to the Amazon S3 bucket configured for your MWAA environment - Permission to update `requirements.txt` and MWAA environment configuration ## Recommended versions{% #recommended-versions %} The following table shows the default versions pinned by MWAA constraints and the recommended upgrade versions for each Airflow version: | Package | Airflow 2.7.2 | Airflow 2.8.1 | Airflow 2.9.2 | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ---------------------------------- | | `apache-airflow-providers-openlineage` | Default: 1.1.0Upgrade: **1.14.0** - [Datadog-patched wheel](https://docs.datadoghq.com/resources/whl/apache_airflow_providers_openlineage-1.14.0-py3-none-any.whl.md) | Default: 1.4.0Upgrade: **1.14.0** | Default: 1.8.0Upgrade:**2.2.0** | | `apache-airflow-providers-common-sql` | Default: 1.7.2Upgrade: no change | Default: 1.10.0Upgrade: **1.20.0** | Default: 1.14.0Upgrade: **1.21.0** | | `apache-airflow-providers-common-compat` | Default: n/aUpgrade: **1.2.2** - [Datadog-patched wheel](https://docs.datadoghq.com/resources/whl/apache_airflow_providers_common_compat-1.2.2-py3-none-any.whl.md) | Default: n/aUpgrade: **1.2.1** | Default: n/aUpgrade: **1.4.0** | | `openlineage-integration-common` | Default: 1.3.1Upgrade: **1.24.2** | Default: 1.7.0Upgrade: **1.24.2** | Default: 1.16.0Upgrade: **1.31.0** | | `openlineage-python` | Default: 1.3.1Upgrade: **1.24.2** | Default: 1.7.0Upgrade: **1.24.2** | Default: 1.16.0Upgrade: **1.31.0** | | `openlineage-sql` | Default: 1.3.1Upgrade: **1.24.2** | Default: 1.7.0Upgrade: **1.24.2** | Default: 1.16.0Upgrade: **1.31.0** | ## Update constraints and requirements{% #update-constraints-and-requirements %} {% tab title="Airflow 2.7.2" %} Airflow 2.7.2 is not compatible with the upstream `apache-airflow-providers-openlineage` 1.14.0 package. Datadog provides patched versions of the provider and `common-compat` packages with relaxed Airflow version constraints for use on MWAA 2.7.2. 1. Download the patched wheel files and upload them to your MWAA S3 bucket: ```shell MWAA_BUCKET= curl -Lo apache_airflow_providers_openlineage-1.14.0-py3-none-any.whl \ "https://docs.datadoghq.com/resources/whl/apache_airflow_providers_openlineage-1.14.0-py3-none-any.whl" curl -Lo apache_airflow_providers_common_compat-1.2.2-py3-none-any.whl \ "https://docs.datadoghq.com/resources/whl/apache_airflow_providers_common_compat-1.2.2-py3-none-any.whl" aws s3 cp apache_airflow_providers_openlineage-1.14.0-py3-none-any.whl "s3://${MWAA_BUCKET}/dags/" aws s3 cp apache_airflow_providers_common_compat-1.2.2-py3-none-any.whl "s3://${MWAA_BUCKET}/dags/" ``` 1. Download the Airflow constraints file: ```shell curl -o constraints.txt \ "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt" ``` **Note**: Check the Python version for your MWAA environment. If your environment uses a different Python version, replace `3.11` in the URL. 1. Edit `constraints.txt` and update these package pins: ```text apache-airflow-providers-openlineage==1.14.0 openlineage-integration-common==1.24.2 openlineage-python==1.24.2 openlineage-sql==1.24.2 ``` Also add the following line, since it is not present in the default constraints file: ```text apache-airflow-providers-common-compat==1.2.2 ``` No change is needed for `apache-airflow-providers-common-sql` (the default `1.7.2` is compatible). 1. Upload the updated constraints file to your MWAA S3 bucket: ```shell aws s3 cp constraints.txt "s3://${MWAA_BUCKET}/dags/constraints.txt" ``` 1. Update `requirements.txt` to reference the patched wheels and constraints: ```text --constraint /usr/local/airflow/dags/constraints.txt /usr/local/airflow/dags/apache_airflow_providers_openlineage-1.14.0-py3-none-any.whl /usr/local/airflow/dags/apache_airflow_providers_common_compat-1.2.2-py3-none-any.whl openlineage-integration-common==1.24.2 openlineage-python==1.24.2 openlineage-sql==1.24.2 ``` 1. Upload the updated `requirements.txt` file: ```shell aws s3 cp requirements.txt "s3://${MWAA_BUCKET}/requirements.txt" ``` {% /tab %} {% tab title="Airflow 2.8.1" %} 1. Download the Airflow constraints file: ```shell curl -o constraints.txt \ "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt" ``` **Note**: Check the Python version for your MWAA environment. If your environment uses a different Python version, replace `3.11` in the URL. 1. Edit `constraints.txt` and update these package pins: ```text apache-airflow-providers-openlineage==1.14.0 apache-airflow-providers-common-sql==1.20.0 openlineage-integration-common==1.24.2 openlineage-python==1.24.2 openlineage-sql==1.24.2 ``` Also add the following line, since it is not present in the default constraints file: ```text apache-airflow-providers-common-compat==1.2.1 ``` 1. Upload the updated constraints file to your MWAA S3 bucket: ```shell MWAA_BUCKET= aws s3 cp constraints.txt "s3://${MWAA_BUCKET}/dags/constraints.txt" ``` 1. Update `requirements.txt` to reference the constraints file and pin the upgraded package versions: ```text --constraint /usr/local/airflow/dags/constraints.txt apache-airflow-providers-openlineage==1.14.0 apache-airflow-providers-common-sql==1.20.0 apache-airflow-providers-common-compat==1.2.1 openlineage-integration-common==1.24.2 openlineage-python==1.24.2 openlineage-sql==1.24.2 ``` 1. Upload the updated `requirements.txt` file: ```shell aws s3 cp requirements.txt "s3://${MWAA_BUCKET}/requirements.txt" ``` {% /tab %} {% tab title="Airflow 2.9.2" %} 1. Download the Airflow constraints file: ```shell curl -o constraints.txt \ "https://raw.githubusercontent.com/apache/airflow/constraints-2.9.2/constraints-3.12.txt" ``` **Note**: Check the Python version for your MWAA environment. If your environment uses a different Python version, replace `3.12` in the URL. 1. Edit `constraints.txt` and update these package pins: ```text apache-airflow-providers-openlineage==2.2.0 apache-airflow-providers-common-sql==1.21.0 openlineage-integration-common==1.31.0 openlineage-python==1.31.0 openlineage-sql==1.31.0 ``` Also add the following line, since it is not present in the default constraints file: ```text apache-airflow-providers-common-compat==1.4.0 ``` 1. Upload the updated constraints file to your MWAA S3 bucket: ```shell MWAA_BUCKET= aws s3 cp constraints.txt "s3://${MWAA_BUCKET}/dags/constraints.txt" ``` 1. Update `requirements.txt` to reference the constraints file and pin the upgraded package versions: ```text --constraint /usr/local/airflow/dags/constraints.txt apache-airflow-providers-openlineage==2.2.0 apache-airflow-providers-common-sql==1.21.0 apache-airflow-providers-common-compat==1.4.0 openlineage-integration-common==1.31.0 openlineage-python==1.31.0 openlineage-sql==1.31.0 ``` 1. Upload the updated `requirements.txt` file: ```shell aws s3 cp requirements.txt "s3://${MWAA_BUCKET}/requirements.txt" ``` {% /tab %} ## Deploy and validate{% #deploy-and-validate %} 1. In the MWAA console, update the environment so it picks up the new `requirements.txt`. 1. Wait for the environment update to complete. 1. Review MWAA install and startup logs. Confirm there are no dependency resolver errors. 1. Trigger a DAG run that emits OpenLineage events. 1. In Datadog, verify that runs appear on the [Data Observability: Jobs Monitoring](https://app.datadoghq.com/data-jobs/) page. ## Troubleshooting{% #troubleshooting %} - If dependency resolution still fails, verify that the same package versions are pinned in both `constraints.txt` and `requirements.txt`. - If MWAA installs older versions, verify that `requirements.txt` points to `--constraint /usr/local/airflow/dags/constraints.txt` and that the file exists in your S3 `dags/` path. - If uploads or reads fail, verify that the MWAA execution role has access to the required S3 objects. For details, see [Amazon MWAA execution role](https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-create-role.html). ## If you manage a custom image{% #if-you-manage-a-custom-image %} If your team manages package versions through a custom Airflow image, update package versions in the image definition first, then regenerate `requirements.txt` from the built image so constraints and requirements stay aligned.