I have a k8s cluster v1.26.4 and all the nodes have crio v1.25.2 installed.
I have private registry with basic auth that mimics as another private registry to reduce code writing. The registry is a docker distribution solution with proxy-mode for latter. Then i have mirror.conf for cri-o as is:
[[registry]]
prefix = private.registry
location = mirror.registry
When i execute
crictl pull --creds private.registry/image:tag
everything is ok. But when i create a pod:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: pod
name: pod
namespace: my-ns
spec:
containers:
- image: private.registry/image:tag
name: pod
imagePullSecrets:
- name: registry-secret
and a secret:
apiVersion: v1
data:
.dockerconfigjson: base64-encoded-auths
kind: Secret
metadata:
name: registry-secret
namespace: my-ns
type: kubernetes.io/dockerconfigjson
then i have this error:
Failed to pull image "private.registry/image:tag": rpc error: code = Unknown desc = reading manifest tag in mirror.registry/image: unauthorized: authentication required
If i rewrite the mirror.conf for cri-o when prefix and location are the same then no issues whatsoever. This means that my auths in a secret are most likely correct.
[[registry]]
prefix = mirror.registry
location = mirror.registry
Turned out direct requests through crictl pull
were not working as well, so the cri-o could be the isse.
After researching in i've found couple of similar questions (this and this) about missing credentials when using redirect(prefix)/mirror with cri-o. The reason is credentials are stripped when named endpoint and the actual are not the same. It is done intentionally to avoid credentials leaking and commented right in a library project. Looks like there is only one solution to use global_auth_file option in crio.conf file, which works.