Search code examples
python-3.xgoogle-cloud-platformairflowpypigoogle-cloud-composer

How to install PYPI package (Facebook-business, Google-cloud-secret-manager) in Google Cloud Composer

I am trying to install Facebook-business, Google-cloud-secret-manager Pypi package in Google Cloud Manager but after 20 min I am getting below error:

enter image description here

enter image description here

Please find the image version as shown above.

Solution

  • Building my test Composer environment with runtime based on composer-1.9.1-airflow-1.10.6 image I've faced with probably the same issue missing PyPI dependencies that belong to google-cloud-secret-manager and facebook_business packages, as well as affecting the system core to properly compile image changes.

    Fortunately, adding the following packages chain to my requirements.txt files permanently fixed the issue:

    google-cloud-secret-manager==1.0.0
grpc-google-iam-v1==0.12.3
google-api-core==1.22.1
googleapis-common-protos==1.52.0
google-cloud-bigtable==1.4.0
google-cloud-spanner==1.17.1
google-auth==1.20.1
setuptools==49.6.0
google-cloud-bigquery==1.27.2
google-cloud-logging==1.15.1
google-cloud-core==1.4.1
protobuf==3.13.0
google-cloud-storage==1.30.0
google-resumable-media==0.7.1
facebook-business==8.0.1
six==1.15.0
mock
pycountry
curlify
requests

    Composer gcloud update command reference guide you can find here.

    FYI. You can catch Composer builder logs via GCP Cloud Build and keep track of the runtime image creation progress.