Why does my container registry work for gitlab autodeploy but not for my custom pipeline?

Question

Hey I'm creating a gitlab pipeline and I have 2 stages: staging and qa. For staging I use gitlabs auto-deploy script and for qa I use a custom script.

With staging the pipeline works. For qa the container registry tells me the username/password is wrong:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Warning  Failed     4m48s (x4 over 6m9s)  kubelet            Failed to pull image "<container-url>": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: HTTP Basic: Access denied
  Warning  Failed     4m48s (x4 over 6m9s)  kubelet            Error: ErrImagePull
  Normal   BackOff    4m8s (x7 over 6m8s)   kubelet            Back-off pulling image "<container-url>"
  Warning  Failed     64s (x20 over 6m8s)   kubelet            Error: ImagePullBackOff

during the pipeline I am creating a secret with login details to the registry. Both pipelines create it with:

  kubectl create secret -n "$KUBE_NAMESPACE" \
    docker-registry "gitlab-registry-${CI_PROJECT_PATH_SLUG}" \
    --docker-server="$CI_REGISTRY" \
    --docker-username="${CI_DEPLOY_USER:-$CI_REGISTRY_USER}" \
    --docker-password="${CI_DEPLOY_PASSWORD:-$CI_REGISTRY_PASSWORD}" \
    --docker-email="$GITLAB_USER_EMAIL" \
    -o yaml --dry-run | kubectl replace -n "$KUBE_NAMESPACE" --force -f -

Since I have no deploy token its defaulting to username=$CI_REGISTRY_USER and password=$CI_REGISTRY_PASSWORD. I double checked and there is a secret in the respective namespaces. Both secrets are the same except for the password, which is generated each time a pipeline is run (as far as I understand).

Finally I'm putting the secret in the deployment file:

spec: 
  template:
    spec:
      imagePullSecrets:
        - <name-of-secret>

So this should give me access to the container registry, but it gives me the error above. Whats the reason it works fine with auto-deploy but not for me? The secret is created the same way so it should yield the same result.

Thanks for your help :)

EDIT:
I added a deploy token under project > settings > repository > deploy tokens and called it gitlab-deploy-token. It has all the rights and its login details are passed to the pipeline via $CI_DEPLOY_USER and $CI_DEPLOY_PASSWORD.

The error in the pod changed to:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  ...
  Warning  Failed     13m (x4 over 15m)  kubelet            Failed to pull image "<image-url>": rpc error: code = Unknown desc = Error reading manifest <container-tag> in <registry-url> errors: denied: requested access to the resource is denied unauthorized: authentication required

According to gitlab documentation I should be able to log in to the registry with buildah login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY. But when I do that in the stage that is just login in and out of the registry I get:

error authenticating creds for "<registry>": pinging docker registry returned: Get <link>: net/http: TLS handshake timeout

with the link being a json file:

errors: 
   0:   
      code:     "UNAUTHORIZED"
      message:  "authentication required"
      detail:   null

Since I'm providing username and password and the error does not complain about them mismatching I'm assuming this is working. But why is the authentication still not going through? Is there something else needed to authenticate? Some certificates perhaps?

The staging pipeline (the one with auto-deploy) which is now also using the gitlab-deploy-token still works.

Answer 1

The secret you generate ($CI_REGISTRY_PASSWORD) is only valid during the specific job. So the moment the job finishes the password is invalid.

Use a deploy token.