How to configure GCR with K3S

Trying to set up Google container registry (GCR) with k3s according to the documentation - Private Registry Configuration https://rancher.com/docs/k3s/latest/en/installation/private-registry/

I keep getting 401 Unauthorized error when running the command crictl pull gcr.io/my-project-id/my-image:latest


WARN[2022-06-08T14:36:38.338217895+03:00] image connect using default endpoints: [unix:///run/k3s/containerd/containerd.sock unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
FATA[2022-06-08T14:36:40.739055570+03:00] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/my-project-id/my-image:latest": failed to resolve reference "gcr.io/my-project-id/my-image:latest": pulling from host gcr.io failed with status code [manifests latest]: 401 Unauthorized

here is my config: /etc/rancher/k3s/registries.yaml

mirrors:
  gcr.io:
    endpoint:
      - "https://gcr.io"
configs:
  "gcr.io":
    auth:
     username: _json_key
     password: |
      {
        "type": "service_account",
        "project_id": "my-project-id",
        "private_key_id": "4c97dc266e4b303fc45dc70561e383ae92ccccae",
        "private_key": "-----BEGIN PRIVATE KEY-----\nXXXIEvAIBADANB........\n-----END PRIVATE KEY-----\n",
        "client_email": "build@my-project-id.iam.gserviceaccount.com",
        "client_id": "108884742074047075648",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/build%40my-project-id.iam.gserviceaccount.com"
      }

Solution

I didn't see it anywhere in the documentation, but I took for granted that the password should be a single line. I suppose I got to that conclusion because is the way I configured it in CI/CD pipelines. Anyway, just convert your password to a single quoted line and you are done (I have tried myself):

mirrors:
  gcr.io:
    endpoint:
      - "https://gcr.io"
configs:
  "gcr.io":
    auth:
     username: _json_key
     password: '{"type": "service_account", "project_id": "my-project-id", "private_key_id": "4c97dc266e4b303fc45dc70561e383ae92ccccae", "private_key": "-----BEGIN PRIVATE KEY-----\nXXXIEvAIBADANB........\n-----END PRIVATE KEY-----\n", "client_email": "build@my-project-id.iam.gserviceaccount.com", "client_id": "108884742074047075648", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/build%40my-project-id.iam.gserviceaccount.com"}'

Then restart k3s.

Please, note you will need to add the registries.yaml file to all your k3s agent nodes and also to your k3s server nodes in case you deploy workloads in them.

Hope this help.