Search code examples
dockerkubernetesartifactory

Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: Bad credentials

We use Artifactory as a hub for all external docker registries. We have different enviornments, all pull form the same url https://docker.intra. We suddenly have one case where a certain image is not pulled anymore but get this error

ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: Bad credentials

Actually

Warning     Failed  Error: ImagePullBackOff     2 minutes ago
Normal  BackOff     Back-off pulling image "docker.intra/main:3.64.0"   5 minutes ago
Normal  Pulling     Pulling image "docker.intra/main:3.64.0"    6 minutes ago
Warning     Failed  Error: ErrImagePull     6 minutes ago
Warning     Failed  Failed to pull image "docker.intra/main:3.64.0": rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: This request is blocked due to recurrent login failures, please try again in 4 seconds    6 minutes ago
Warning     Failed  Failed to pull image "docker.intra/main:3.64.0": rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: Bad credentials   6 minutes ago
Warning     Failed  Failed to pull image "docker.intra/main:3.64.0": rpc error: code = Unknown desc = Error response from daemon: Get https://docker.intra/v2/main/manifests/3.64.0: unknown: This request is blocked due to recurrent login failures, please try again in 3 seconds    7 minutes ago
Normal  Scheduled   Successfully assigned stackrox/central-6487fdc867-jq4j5 to k8s0001  7 minutes ago

I however, can successfully download the image from within the mentioned node k8s0001

[papanito@k8s0001 ~]$ docker pull docker.intra/main:3.64.0

    3.64.0: Pulling from main
    29291e31a76a: Pulling fs layer
    ...
    bcabcd2816e8: Pull complete
    Digest: sha256:d03c7cf13b296d8e75529651edae6f08c56144bc090ad79baddccebc07c5a491
    Status: Downloaded newer image for docker.intra/main:3.64.0

There is no authentication defined and should not be required. Here the daemon.json

{
    "insecure-registries": null,
    "log-driver": "json-file",
    "data-root": "/mnt/data/docker",
    "log-opts": {
    "max-size": "50m"
    },
    "log-level": "info",
    "storage-driver": "overlay2",
    "icc": false,
    "registry-mirrors": [
    "https://docker.intra"
    ],
    "userland-proxy": false,
    "no-new-privileges": false,
    "live-restore": true
}

I wonder where the "Bad Credentials" come from and why other images in the same cluster are downloaded successfully!?

Solution

  • The root cause for this behavior is not clear, however it seems it's related to the namespace. Pulling the docker image within another namespace works fine. Same works if one deploys the application in a new namespace.

    Alternatively you can also delete the ns completely and then re-create it.