After deleting image from Google Container Registry and uploading another with the same tag, deleted one is still pulled

Question

I don't know if this is intended behavior or bug in GCR. Basically I tried do it like that:

1. Create image from local files using Docker on Windows (Linux based image).
2. Before creating image I delete all local images with the same name/tag.
3. Image is tagged like repostiory/project/name:v1
4. When testing locally image have correct versions of executables (docker run imageID).
5. Before pushing image to GCR I delete all images from GCR with the same tag/name.
6. When Trying to pull new image from GCR to example kubernetes it pull the first (ever) image uploaded under particular tag.

I want to reuse the same tag to not change config file with every test and I don't really need to store previous versions of images.

Answer 1

It sounds like you're hitting the problem described in kubernetes/kubernetes#42171.

tl;dr, the default pull policy of kubernetes is broken by design such that you cannot reuse tags (other than latest). I believe the guidance from the k8s community is to use "immutable tags", which is a bit of an oxymoron.

You have a few options:

1. Switch to using the latest tag, since kubernetes has hardcoded this in their default pull policy logic (I believe in an attempt to mitigate the problem you're having).
2. Never reuse a tag.
3. Switch to explicitly using the PullAlways ImagePullPolicy. If you do this, you will incur a small overhead, since your node will have to check with the registry that the tag has not changed.
4. Switch to deploying by image digest with the PullIfNotPresent ImagePullPolicy. A more detailed explanation is in the PR I linked, but this gets you the best of both worlds.