Search code examples
dockergoogle-compute-enginegoogle-kubernetes-enginegoogle-container-registry

How to pull/push from/to GCR from GKE node

I'm building an application that I will run in GKE. This application will use shell commands (for now) to build docker images and try to push them to GCR. I'm finding that when I try to do this from a pod running in GKE I get authentication problems. I'm having trouble figuring out why these authentication problems are happening.

Here's a list of all of the debugging I've done so far. At the highest level, my GKE clusters have the https://www.googleapis.com/auth/devstorage.read_write oauth scope. When I examine the permissions on the underlying GCE instance, I see these permissions - note the Read Write value for Storage:

enter image description here

Now, when I SSH into that instance using the console and list the docker images I see the image used by GKE when spinning up my pod:

paymahn@gke-prod-478557c-default-pool-e9314f46-d9mn ~ $ docker image ls
REPOSITORY                                                              TAG                 IMAGE ID            CREATED             SIZE
gcr.io/gadic-310112/server                                              latest              8f8a22237c31        2 days ago          1.85GB
...

However, if I try to manually pull that image while SSH-ed into the GCP instance, I get an authentication problem:

paymahn@gke-prod-478557c-default-pool-e9314f46-d9mn ~ $ docker pull gcr.io/gadic-310112/server:latest
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

I also looked at the service account 65106360748-compute@developer.gserviceaccount.com which is the default compute instance service account. Here are the permissions it has (I manually added the Storage Object Creator role):

enter image description here

Adding the Storage Object Creator role to that service account didn't help.

Is my approach to authentication here fundamentally flawed? It seems like I have all the right pieces in place to pull/push from GCR from GKE. Maybe there's an extra step I need to do for the docker client to authenticate?

Solution

  • Figured it out. I had to:

    1. make a service account with the roles/storage.objectAdmin
    2. generate a key for that service account
    3. store that key as a secret in GKE
    4. Mount that secret into my pods
    5. run gcloud auth activate-service-account --key-file <path to key>
    6. run gcloud auth configure-docker

    Once all of that was done, my pods could pull from and push to GCR.