Installing pypi package google-cloud-datastore on Google Cloud Composer fails

Question

I'm running Airflow 1.10.6 on Google Cloud Composer with Python 3. To initiate a dataflow job from the composer environment I need to have the PyPi package google-cloud-datastore. When I try to add this package through either the interface or cloud shell, the build fails. It raises the following error:

UPDATE operation on this environment failed 1 hour ago with the following error message: Failed to install PyPI packages.

If I check the logs of Google Build for the Kubernetes Engine I see the following error:

I 2020-03-04T14:36:23.939075607Z [0mgoogle-cloud-datastore 1.11.0 has requirement google-cloud-core<2.0dev,>=1.0.3, but you have google-cloud-core 0.29.1.

I 2020-03-04T14:36:23.939669242Z google-cloud-datastore 1.11.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 1.8.1.

So google-cloud-datastore requires google-cloud-core>=1.0.3. I tried to explicitly set google-cloud-core to 1.0.3, but then I get the following error:

I 2020-03-05T08:13:18.539300693Z [0mgoogle-cloud-logging 1.9.1 has requirement google-cloud-core<0.30dev,>=0.29.0, but you have google-cloud-core 1.0.3.

I 2020-03-05T08:13:18.539878367Z google-cloud-bigtable 0.32.0 has requirement google-cloud-core<0.30dev,>=0.29.0, but you have google-cloud-core 1.0.3.

I 2020-03-05T08:13:18.540308937Z google-cloud-bigquery 1.8.1 has requirement google-cloud-core<0.30dev,>=0.29.0, but you have google-cloud-core 1.0.3.

So the ranges of versions required for the packages are not overlapping. Does anyone know if this can be fixed? Please let me know.

Answer 1

There is a compatibility issue with google-cloud-datastore in the main container of Composer, and that the latest version of this package could not be installed. The Composer product team has confirmed the issue and is working on it.

Attempting to install an older version of google-cloud-datastore may be a workaround before the root cause is fixed. I have tried with 1.7.4 version and got positive result:

I hope it helps.