Search code examples
dockergoogle-cloud-platformpipgoogle-cloud-buildcloudbuild.yaml

Cannot install private dependency from artifact registry inside docker build

I am trying to install a private python package that was uploaded to an artifact registry inside a docker container (to deploy it on cloudrun).

I have sucessfully used that package in a cloud function in the past, so I am sure the package works.

cloudbuild.yaml

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']

Dockerfile

FROM python:3.8.6-slim-buster

ENV APP_PATH=/usr/src/app
ENV PORT=8080

# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 

RUN pip install --upgrade pip

# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH

COPY / $APP_PATH

WORKDIR $APP_PATH

RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs


# Expose port 
EXPOSE $PORT

# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

The permissions I added are:

  • cloudbuild default service account ([email protected]): Artifact Registry Reader
  • service account running the cloudbuild : Artifact Registry Reader
  • service account running the app: Artifact Registry Reader

The cloudbuild error:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
Solution

  • From your traceback log, we can see that Cloud Build doesn't have the credentials to authenticate to the private repo:

    Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME

    I uploaded a simple package to a private Artifact Registry repo to test this out when building a container and also received the same message. Since you seem to be authenticating with a service account key, the username and password will need to be stored inside pip.conf:

    pip.conf

    [global]
extra-index-url = https://_json_key_base64:[email protected]/PROJECT/REPOSITORY/simple/

    This file therefore needs to be available during the build process. Multi-stage docker builds are very useful here to ensure the configuration keys are not exposed, since we can choose what files make it into the final image (configuration keys would only be present while used to download the packages from the private repo):

    Sample Dockerfile

    # Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build

# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf

WORKDIR /packages/
COPY requirements.txt /

# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf

# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES

# Final image that will be deployed
FROM python:3.8.6-slim-buster

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app

WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/

# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*

WORKDIR $APP_HOME
COPY ./src/main.py ./

# Executing sample flask web app 
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

    I based the dockerfile above on this related thread, and I could confirm the packages were correctly downloaded from my private Artifact Registry repo, and also that the pip.conf file was not present in the resulting image.