Search code examples
djangogoogle-cloud-platformgoogle-cloud-rungoogle-cloud-buildrequirements.txt

Google Cloud Run for Django, Cloud Build fails for psycop2-binary


I have been successfully using Google Build for continuous integration with Google Cloud Run for the Django Application.

However recently psycop2-binary started giving errors as below

Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 202.7/202.7 kB 

28.3 MB/s eta 0:00:00
Step #0 - "Buildpack": Collecting google-cloud-build==3.9.0
Step #0 - "Buildpack":   Downloading google_cloud_build-3.9.0-py2.py3-none-any.whl (88 kB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.2/88.2 kB 13.0 MB/s eta 0:00:00
Step #0 - "Buildpack": Collecting psycopg2-binary==2.9.3
Step #0 - "Buildpack":   Downloading psycopg2-binary-2.9.3.tar.gz (380 kB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 380.6/380.6 kB 39.3 MB/s eta 0:00:00
Step #0 - "Buildpack":   Preparing metadata (setup.py): started
Step #0 - "Buildpack":   Preparing metadata (setup.py): finished with status 'error'
Step #0 - "Buildpack":   error: subprocess-exited-with-error
Step #0 - "Buildpack":   
Step #0 - "Buildpack":   × python setup.py egg_info did not run successfully.
Step #0 - "Buildpack":   │ exit code: 1
Step #0 - "Buildpack":   ╰─> [25 lines of output]
Step #0 - "Buildpack":       /layers/google.python.runtime/python/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
Step #0 - "Buildpack":         warnings.warn(msg, warning_class)
Step #0 - "Buildpack":       running egg_info
Step #0 - "Buildpack":       creating /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info
Step #0 - "Buildpack":       writing /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/PKG-INFO
Step #0 - "Buildpack":       writing dependency_links to /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/dependency_links.txt
Step #0 - "Buildpack":       writing top-level names to /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/top_level.txt
Step #0 - "Buildpack":       writing manifest file '/tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/SOURCES.txt'
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       Error: pg_config executable not found.
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       pg_config is required to build psycopg2 from source.  Please add the directory
Step #0 - "Buildpack":       containing pg_config to the $PATH or specify the full executable path with the
Step #0 - "Buildpack":       option:
Step #0 - "Buildpack":       
Step #0 - "Buildpack":           python setup.py build_ext --pg-config /path/to/pg_config build ...
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       or with the pg_config option in 'setup.cfg'.
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       If you prefer to avoid building psycopg2 from source, please install the PyPI
Step #0 - "Buildpack":       'psycopg2-binary' package instead.
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       For further information please check the 'doc/src/install.rst' file (also at
Step #0 - "Buildpack":       <https://www.psycopg.org/docs/install.html>).
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       [end of output]
Step #0 - "Buildpack":   
Step #0 - "Buildpack":   note: This error originates from a subprocess, and is likely not a problem with pip.
Step #0 - "Buildpack": error: metadata-generation-failed
Step #0 - "Buildpack": 
Step #0 - "Buildpack": × Encountered error while generating package metadata.
Step #0 - "Buildpack": ╰─> See above for output.
Step #0 - "Buildpack": 
Step #0 - "Buildpack": note: This is an issue with the package mentioned above, not pip.
Step #0 - "Buildpack": hint: See above for details.
Step #0 - "Buildpack": Done "python3 -m pip install --requirement requirements.txt --upgr..." (5.263140792s)
Step #0 - "Buildpack": Failure: (ID: c84b3231) ...bernetes==24.2.0
Step #0 - "Buildpack":   Downloading kubernetes-24.2.0-py2.py3-none-any.whl (1.5 MB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 72.6 MB/s eta 0:00:00
Step #0 - "Buildpack": Collecting google-cloud-container==2.11.0
Step #0 - "Buildpack":   Downloading google_cloud_container-2.11.0-py2.py3-none-any.whl (202 kB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 202.7/202.7 kB 28.3 MB/s eta 0:00:00
Step #0 - "Buildpack": Collecting google-cloud-build==3.9.0
Step #0 - "Buildpack":   Downloading google_cloud_build-3.9.0-py2.py3-none-any.whl (88 kB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.2/88.2 kB 13.0 MB/s eta 0:00:00
Step #0 - "Buildpack": Collecting psycopg2-binary==2.9.3
Step #0 - "Buildpack":   Downloading psycopg2-binary-2.9.3.tar.gz (380 kB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 380.6/380.6 kB 39.3 MB/s eta 0:00:00
Step #0 - "Buildpack":   Preparing metadata (setup.py): started
Step #0 - "Buildpack":   Preparing metadata (setup.py): finished with status 'error'
Step #0 - "Buildpack":   error: subprocess-exited-with-error
Step #0 - "Buildpack":   
Step #0 - "Buildpack":   × python setup.py egg_info did not run successfully.
Step #0 - "Buildpack":   │ exit code: 1
Step #0 - "Buildpack":   ╰─> [25 lines of output]
Step #0 - "Buildpack":       /layers/google.python.runtime/python/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
Step #0 - "Buildpack":         warnings.warn(msg, warning_class)
Step #0 - "Buildpack":       running egg_info
Step #0 - "Buildpack":       creating /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info
Step #0 - "Buildpack":       writing /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/PKG-INFO
Step #0 - "Buildpack":       writing dependency_links to /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/dependency_links.txt
Step #0 - "Buildpack":       writing top-level names to /tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/top_level.txt
Step #0 - "Buildpack":       writing manifest file '/tmp/pip-pip-egg-info-a97zvtp4/psycopg2_binary.egg-info/SOURCES.txt'
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       Error: pg_config executable not found.
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       pg_config is required to build psycopg2 from source.  Please add the directory
Step #0 - "Buildpack":       containing pg_config to the $PATH or specify the full executable path with the
Step #0 - "Buildpack":       option:
Step #0 - "Buildpack":       
Step #0 - "Buildpack":           python setup.py build_ext --pg-config /path/to/pg_config build ...
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       or with the pg_config option in 'setup.cfg'.
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       If you prefer to avoid building psycopg2 from source, please install the PyPI
Step #0 - "Buildpack":       'psycopg2-binary' package instead.
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       For further information please check the 'doc/src/install.rst' file (also at
Step #0 - "Buildpack":       <https://www.psycopg.org/docs/install.html>).
Step #0 - "Buildpack":       
Step #0 - "Buildpack":       [end of output]
Step #0 - "Buildpack":   
Step #0 - "Buildpack":   note: This error originates from a subprocess, and is likely not a problem with pip.
Step #0 - "Buildpack": error: metadata-generation-failed
Step #0 - "Buildpack": 
Step #0 - "Buildpack": × Encountered error while generating package metadata.
Step #0 - "Buildpack": ╰─> See above for output.
Step #0 - "Buildpack": 
Step #0 - "Buildpack": note: This is an issue with the package mentioned above, not pip.
Step #0 - "Buildpack": hint: See above for details.
Step #0 - "Buildpack": --------------------------------------------------------------------------------
Step #0 - "Buildpack": Sorry your project couldn't be built.
Step #0 - "Buildpack": Our documentation explains ways to configure Buildpacks to better recognise your project:
Step #0 - "Buildpack":  -> https://github.com/GoogleCloudPlatform/buildpacks/blob/main/README.md
Step #0 - "Buildpack": If you think you've found an issue, please report it:
Step #0 - "Buildpack":  -> https://github.com/GoogleCloudPlatform/buildpacks/issues/new
Step #0 - "Buildpack": --------------------------------------------------------------------------------
Step #0 - "Buildpack": ERROR: failed to build: exit status 1
Step #0 - "Buildpack": ERROR: failed to build: executing lifecycle: failed with status code: 51
Finished Step #0 - "Buildpack"
ERROR
ERROR: build step 0 "gcr.io/k8s-skaffold/pack" failed: step exited with non-zero status: 1

You can find the requirements information as below as well :

Django==3.1.0
google-auth
google-cloud
google-cloud-pubsub==2.5.0
google-cloud-storage==1.25.0
google-cloud-secret-manager==0.1.1
django-sitemap-generate==0.5.0
django-cors-headers==3.2.1
djangorestframework==3.11.1
django-storages==1.9.1
django-anymail[mailgun]
django-robots==4.0
python-dateutil==2.8.1
gunicorn==20.0.4
requests==2.23.0
stripe==2.43.0
six==1.14.0
google-api-python-client==1.8.3
pyparsing==2.4.7
protobuf==3.20.*
djangorestframework-jwt==1.10.0
djangorestframework-simplejwt==5.2.0
django-filter==21.1
google-cloud-logging>=3.0.0
kubernetes==24.2.0
google-cloud-container==2.11.0
google-cloud-build==3.9.0
psycopg2-binary==2.9.3

How can I fix this issue ?

Update :

Pip version is already 22.3.1.

Step #0 - "Buildpack": Running "sed -i s|/opt/python3.11|/layers/google.python.runtime/python|g /layers/google.python.runtime/python/bin/2to3-3.11"
Step #0 - "Buildpack": Done "sed -i s|/opt/python3.11|/layers/google.python.runtime/pytho..." (1.28464ms)
Step #0 - "Buildpack": Upgrading pip to the latest version and installing build tools
Step #0 - "Buildpack": --------------------------------------------------------------------------------
Step #0 - "Buildpack": Running "/layers/google.python.runtime/python/bin/python3 -m pip install --upgrade pip setuptools==v64.0.0 wheel"
Step #0 - "Buildpack": Requirement already satisfied: pip in /layers/google.python.runtime/python/lib/python3.11/site-packages (22.3)
Step #0 - "Buildpack": Collecting pip
Step #0 - "Buildpack":   Downloading pip-22.3.1-py3-none-any.whl (2.1 MB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 25.2 MB/s eta 0:00:00
Step #0 - "Buildpack": Collecting setuptools==v64.0.0
Step #0 - "Buildpack":   Downloading setuptools-64.0.0-py3-none-any.whl (1.2 MB)
Step #0 - "Buildpack":      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 80.9 MB/s eta 0:00:00

Solution

  • In addition to your answer @london_utku for new feature in psycopg 2.9.5

    • Add support for Python 3.11.
    • Add support for rowcount in MERGE statements in binary packages (ticket #1497).
    • Wheel package compiled against OpenSSL 1.1.1r and PostgreSQL 15 libpq.

    You can also review all the Release notes for Psycopg 2.9.5 documentation