gcloud docker authentication x509 error

I am getting authentication errors when trying to push to google container repository. I am seeing a lot of questions on this, and will go through the steps i've done based on additional suggestions. I am working locally on Windows, with a up to date version of gcloud and docker.

Let's start that I have an active service account with full owner permissions.

I can see into my project and bucket just fine.

C:\Program Files (x86)\Google\Cloud SDK>gsutil acl get gs://api-project-773889352370-ml
[
  {
    "entity": "project-owners-773889352370",
    "projectTeam": {
      "projectNumber": "773889352370",
      "team": "owners"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-editors-773889352370",
    "projectTeam": {
      "projectNumber": "773889352370",
      "team": "editors"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-viewers-773889352370",
    "projectTeam": {
      "projectNumber": "773889352370",
      "team": "viewers"
    },
    "role": "READER"
  }
]

I can download a .json key for my service account and activate it locally.

gcloud auth activate-service-account --key-file C:/Users/Ben/Dropbox/Google/MeerkatReader-9fbf10d1e30c.json

I have a working set of docker and logins

C:\Program Files (x86)\Google\Cloud SDK>docker push bw4sz/hello-world
The push refers to a repository [docker.io/bw4sz/hello-world]
a02596fdd012: Layer already exists
latest: digest: sha256:a18ed77532f6d6781500db650194e0f9396ba5f05f8b50d4046b294ae5f83aa4 size: 524

I tag my test image (hello-world) with the hostname and my project ID and give it a try

C:\Program Files (x86)\Google\Cloud SDK>gcloud docker --push gcr.io/api-project-773889352370/hello-world

ERROR: Docker CLI operation failed:

Error response from daemon: Get https://gcr.kubernetes.io/v1/users/: x509: certificate has expired or is not yet valid

ERROR: (gcloud.docker) Docker login failed.

Fine, let's go to advanced auth methods suggested here, here, and here

The documentation says I can bypass gcloud and use docker directly, just pass my .json keyfile to login.

C:\Program Files (x86)\Google\Cloud SDK>docker login -u _json_key -p "$(cat C:/Users/Ben/Dropbox/Google/MeerkatReader-d77c0d6aa04f.json)" https://gcr.io
Error response from daemon: Get https://gcr.io/v2/: unknown: Unable to parse json key.

The docs say to do this in two steps, let's try that:

C:\Program Files (x86)\Google\Cloud SDK>set /p PASS=<C:/Users/Ben/Dropbox/Google/MeerkatReader-9fbf10d1e30c.json
C:\Program Files (x86)\Google\Cloud SDK>docker login -e [email protected] -u _json_key -p "%PASS%" https://gcr.io
Flag --email has been deprecated, will be removed in 1.13.
Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password

Another suggestion was to use a temp token

C:\Program Files (x86)\Google\Cloud SDK>docker login -e [email protected] -u _token -p "$(gcloud auth print-access-token)" https://gcr.io
Flag --email has been deprecated, will be removed in 1.13.
Login Succeeded

Great let's try to push again.

C:\Program Files (x86)\Google\Cloud SDK>docker push gcr.io/api-project-773889352370/hello-world
The push refers to a repository [gcr.io/api-project-773889352370/hello-world]
a02596fdd012: Preparing
denied: Unable to create the repository, please check that you have access to do so.

I saw someone suggest to just push to the bucket and not the project ID, that just hangs

C:\Program Files (x86)\Google\Cloud SDK>docker push gcr.io/api-project-773889352370-ml/hello-world
The push refers to a repository [gcr.io/api-project-773889352370-ml/hello-world]
a02596fdd012: Retrying in 1 second <- goes on forever.

EDIT: Trying suggestion solution

C:\Program Files (x86)\Google\Cloud SDK>gcloud auth activate-service-account --key-file C:/Users/Ben/Dropbox/Google/MeerkatReader-9fbf10d1e30c.json
Activated service account credentials for: [[email protected]]

C:\Program Files (x86)\Google\Cloud SDK>gcloud docker -a -s gcr.io
Short-lived access for ['gcr.io'] configured.

C:\Program Files (x86)\Google\Cloud SDK>docker push gcr.io/api-project-773889352370/hello-world
The push refers to a repository [gcr.io/api-project-773889352370/hello-world]
a02596fdd012: Preparing
denied: Unable to create the repository, please check that you have access to do so.

Solution

Its a bit of a hack, but the only solution I can find is to

Push image to dockerhub
Spin up an compute engine instance
install docker (careful of which version, I had 1.6, I'll need to go > 1.9 in the future, according the GCR docs)
Pull image from dockerhub
gcloud docker push to GCR

That worked for me. Not a great solution. I welcome suggestions.