Search code examples
airflowgoogle-cloud-composergoogle-cloud-data-fusion

Issue installing apache-airflow-backport-providers-google module on airflow instance of Google Composer

I need execute Data Fusion pipelines from Composer, using de operatos for this:

from airflow.providers.google.cloud.operators.datafusion import (
    CloudDataFusionCreateInstanceOperator,
    CloudDataFusionCreatePipelineOperator,
    CloudDataFusionDeleteInstanceOperator,
    CloudDataFusionDeletePipelineOperator,
    CloudDataFusionGetInstanceOperator,
    CloudDataFusionListPipelinesOperator,
    CloudDataFusionRestartInstanceOperator,
    CloudDataFusionStartPipelineOperator,
    CloudDataFusionStopPipelineOperator,
    CloudDataFusionUpdateInstanceOperator,
)

The issue I have is about modulo "apache-airflow-backport-providers-google", with the support of this links i knew what I need to use this modulo:

  1. reference to install the modulo in airflow instance (answered by @Gonzalo Pérez Fernández): https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/datafusion.html

when i tried to install python dependency on Composer like PyPi Package i get this error:

UPDATE operation on this environment failed 7 minutes ago with the following error message:
Failed to install PyPI packages. 
apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0. Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/a2ecf37a-4c47-4770-9489-6fb65e87d82f?project=341768372632 for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installation

the log deail is:

apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.
apache-airflow-backport-providers-google 2021.3.3 has requirement apache-airflow~=1.10, but you have apache-airflow 2.1.2+composer.
The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION  fail' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

is there any way to use de module "apache-airflow-backport-providers-google" without depedency issues on composer instance?, or What would be the best way to use data fusion operators no need to change or parse package versions in python?.

Composer Image version used: composer-1.17.0-airflow-2.1.2

Thanks.

Solution

  • There is no need to install apache-airflow-backport-providers-google in Airflow 2.0+. This package actually backports Airflow 2 operators into Airflow 1.10.*. In addition, in Composer version composer-1.17.0-airflow-2.1.2 the apache-airflow-providers-google==5.0.0 package is already installed according to the documentation. You should be able to import the Data Fusion operators with the code snippet you posted as is.

    However, if this is not the case, you should probably handle the conflict shown in the logs when trying to reinstall apache-airflow-providers-google==5.0.0:

    apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.

    You can add the requirement for google-ads=12.0.0 in your PyPi dependencies and see if it works.