Getting error when adding PyPI package to Composer environment

Question

I just created a new Cloud Composer environment and I'm getting an error when trying to add some PyPI packages to the environment (for example google-cloud-pubsub).

This is the data about the environment:

The error I get when trying to add google-cloud-pubsub is just this: Failed to install PyPI packages.

Does anyone know about this?

Thanks :)

Cloud build:

+ python3 -m pip install -r requirements.txt
Collecting google-cloud-pubsub==1.2.0 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/c5/9e/e95b75789ba589e022f856754da8c6a7e7d607861254c7fd8ccdf5d7bdac/google_cloud_pubsub-1.2.0-py2.py3-none-any.whl (121kB)
Requirement already satisfied: apache-airflow==1.10.2-composer in /usr/local/lib/airflow (from -r requirements.txt (line 2))
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0 (from google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/63/7e/a523169b0cc9ce62d56e07571db927286a94b1a5f51ac220bd97db825c77/google_api_core-1.16.0-py2.py3-none-any.whl (70kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3 (from google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/65/19/2060c8faa325fddc09aa67af98ffcb6813f39a0ad805679fa64815362b3a/grpc-google-iam-v1-0.12.3.tar.gz
Requirement already satisfied: alembic<0.9,>=0.8.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: configparser<3.6.0,>=3.5.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: croniter<0.4,>=0.3.17 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: dill<0.3,>=0.2.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: enum34~=1.1.6 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask<0.13,>=0.12.4 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-appbuilder==1.12.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-admin==1.5.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-caching<1.4.0,>=1.3.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-login<0.5,>=0.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-swagger==0.2.13 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: flask-wtf<0.15,>=0.14.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: funcsigs==1.0.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: future<0.17,>=0.16.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: gitpython>=2.0.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: gunicorn<20.0,>=19.4.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: iso8601>=0.1.12 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: jsonschema~=3.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: json-merge-patch==0.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: jinja2<=2.10.0,>=2.7.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: markdown<3.0,>=2.5.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pandas<1.0.0,>=0.17.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pendulum==1.4.4 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: psutil<6.0.0,>=4.2.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pygments<3.0,>=2.0.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-daemon<2.2,>=2.1.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-dateutil<3,>=2.3 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-nvd3==0.15.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: requests<3,>=2.20.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: setproctitle<2,>=1.1.8 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: sqlalchemy<1.3.0,>=1.1.15 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: tabulate<=0.8.2,>=0.7.5 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: tenacity==4.8.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: thrift>=0.9.2 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: tzlocal>=1.4 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: unicodecsv>=0.14.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: werkzeug<0.15.0,>=0.14.1 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /opt/python3.6/lib/python3.6/site-packages (from apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Collecting googleapis-common-protos<2.0dev,>=1.6.0 (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/05/46/168fd780f594a4d61122f7f3dc0561686084319ad73b4febbf02ae8b32cf/googleapis-common-protos-1.51.0.tar.gz
Requirement already satisfied: six>=1.10.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: pytz in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: google-auth<2.0dev,>=0.4.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: setuptools>=34.0.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: protobuf>=3.4.0 in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: grpcio<2.0dev,>=1.8.2; extra == "grpc" in /opt/python3.6/lib/python3.6/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: python-editor>=0.3 in /opt/python3.6/lib/python3.6/site-packages (from alembic<0.9,>=0.8.3->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Mako in /opt/python3.6/lib/python3.6/site-packages (from alembic<0.9,>=0.8.3->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: itsdangerous>=0.21 in /opt/python3.6/lib/python3.6/site-packages (from flask<0.13,>=0.12.4->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: click>=2.0 in /opt/python3.6/lib/python3.6/site-packages (from flask<0.13,>=0.12.4->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: colorama<1,>=0.3.9 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Flask-Babel<1,>=0.11.1 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.3 in /opt/python3.6/lib/python3.6/site-packages (from flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: wtforms in /opt/python3.6/lib/python3.6/site-packages (from flask-admin==1.5.2->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: PyYAML>=3.0 in /opt/python3.6/lib/python3.6/site-packages (from flask-swagger==0.2.13->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: ordereddict in /opt/python3.6/lib/python3.6/site-packages (from funcsigs==1.0.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: gitdb2>=2.0.0 in /opt/python3.6/lib/python3.6/site-packages (from gitpython>=2.0.2->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: attrs>=17.4.0 in /opt/python3.6/lib/python3.6/site-packages (from jsonschema~=3.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/python3.6/lib/python3.6/site-packages (from jsonschema~=3.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: MarkupSafe>=0.23 in /opt/python3.6/lib/python3.6/site-packages (from jinja2<=2.10.0,>=2.7.3->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: numpy>=1.12.0 in /opt/python3.6/lib/python3.6/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pytzdata>=2018.3.0.0 in /opt/python3.6/lib/python3.6/site-packages (from pendulum==1.4.4->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: lockfile>=0.10 in /opt/python3.6/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: docutils in /opt/python3.6/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: python-slugify>=1.2.5 in /opt/python3.6/lib/python3.6/site-packages (from python-nvd3==0.15.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: idna<2.9,>=2.5 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: certifi>=2017.4.17 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/python3.6/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: monotonic>=0.6 in /opt/python3.6/lib/python3.6/site-packages (from tenacity==4.8.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: rsa>=3.1.4 in /opt/python3.6/lib/python3.6/site-packages (from google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: cachetools>=2.0.0 in /opt/python3.6/lib/python3.6/site-packages (from google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/python3.6/lib/python3.6/site-packages (from google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: python3-openid>=2.0 in /opt/python3.6/lib/python3.6/site-packages (from Flask-OpenID<2,>=1.2.5->flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: Babel>=2.3 in /opt/python3.6/lib/python3.6/site-packages (from Flask-Babel<1,>=0.11.1->flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: smmap2>=2.0.0 in /opt/python3.6/lib/python3.6/site-packages (from gitdb2>=2.0.0->gitpython>=2.0.2->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: text-unidecode==1.2 in /opt/python3.6/lib/python3.6/site-packages (from python-slugify>=1.2.5->python-nvd3==0.15.0->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Requirement already satisfied: pyasn1>=0.1.3 in /opt/python3.6/lib/python3.6/site-packages (from rsa>=3.1.4->google-auth<2.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-pubsub==1.2.0->-r requirements.txt (line 1))
Requirement already satisfied: defusedxml in /opt/python3.6/lib/python3.6/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->flask-appbuilder==1.12.1->apache-airflow==1.10.2-composer->-r requirements.txt (line 2))
Building wheels for collected packages: grpc-google-iam-v1, googleapis-common-protos
  Running setup.py bdist_wheel for grpc-google-iam-v1: started
  Running setup.py bdist_wheel for grpc-google-iam-v1: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/de/3a/83/77a1e18e1a8757186df834b86ce6800120ac9c79cd8ca4091b
  Running setup.py bdist_wheel for googleapis-common-protos: started
  Running setup.py bdist_wheel for googleapis-common-protos: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/2c/f9/7f/6eb87e636072bf467e25348bbeb96849333e6a080dca78f706
Successfully built grpc-google-iam-v1 googleapis-common-protos
Installing collected packages: googleapis-common-protos, google-api-core, grpc-google-iam-v1, google-cloud-pubsub
  Found existing installation: googleapis-common-protos 1.5.8
    Uninstalling googleapis-common-protos-1.5.8:
      Successfully uninstalled googleapis-common-protos-1.5.8
  Found existing installation: google-api-core 1.8.1
    Uninstalling google-api-core-1.8.1:
      Successfully uninstalled google-api-core-1.8.1
  Found existing installation: grpc-google-iam-v1 0.11.4
    Uninstalling grpc-google-iam-v1-0.11.4:
      Successfully uninstalled grpc-google-iam-v1-0.11.4
Successfully installed google-api-core-1.16.0 google-cloud-pubsub-1.2.0 googleapis-common-protos-1.51.0 grpc-google-iam-v1-0.12.3
You are using pip version 9.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
+ python3 -m pip check
google-cloud-spanner 1.8.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.
google-cloud-bigtable 0.32.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.
You are using pip version 9.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

Answer 1

The error "Failed to install PyPI packages" means Cloud Build was unable to build Docker images for your Cloud Composer environment. This is usually because there was a PyPI package installation failure, or there were incompatible dependencies. In your case, it is the latter:

+ python3 -m pip check
google-cloud-spanner 1.8.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.
google-cloud-bigtable 0.32.0 has requirement grpc-google-iam-v1<0.12dev,>=0.11.4, but you have grpc-google-iam-v1 0.12.3.

To check for incompatible dependencies (and to prevent them from breaking your environment), Composer runs pip check at the end of all package installs. You will need to resolve these conflicts for the environment update to succeed.