Search code examples
dockergitlabgitlab-cicicdargocd

ArgoCD: no available registry endpoint: failed to fetch anonymous token: unexpected status: 403 Forbidden

Context

A project is worked in a single GitLab repository, where each subproject contains its src/ folder, Dockerfile, etc. A similar structure is followed:

|_ sub-proj_1/
|_ sub-proj_2/
|_ sub-proj_n/
|_ docker-compose.yml
|_ .gitlab-ci.yml
|_ README.md

When the pipeline is launched, it is filtered by sub-project and the image is built with Kaniko, which is sent to the GitLab registry adding a custom path. The PROJECT variable allows to control sub-projects, i.e. sub_proj_1/, sub_proj_2/, sub_proj_n/ in the Gitlab registry.

publish_image:
  stage: publish_image
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  before_script:
    - PROJECT=$(echo "$CI_MERGE_REQUEST_TITLE" | grep -o -w -e "api" -e "frontend" -e "db")
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - echo "PROJECT=$PROJECT" >> build.env
    - /kaniko/executor --context $PROJECT --dockerfile $PROJECT/Dockerfile --destination ${CI_REGISTRY_IMAGE}/${PROJECT}:${CI_COMMIT_SHORT_SHA}
  artifacts:
    reports:
      dotenv: build.env
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH && ($CI_MERGE_REQUEST_TITLE =~ /.*api.*/ || $CI_MERGE_REQUEST_TITLE =~ /.*frontend.*/ || $CI_MERGE_REQUEST_TITLE =~ /.*db.*/)
      when: always

Images are uploaded to the GitLab registry without problems:

enter image description here

There are custom paths in the GitLab registry associated with each subproject:

enter image description here

Gitlab container naming convention for container images supports this kind of custom path.

The cluster has been added via: argocd repo add https://git.xxx/xxx/xxx/xxx.git --username xxx --password xxx

enter image description here

Issue

The images are downloaded to a staging environment using a GitOps controller such as ArgoCD and this error is reported:

enter image description here

Troubleshooting

  1. It has been tested with a registry where the path is not customized and the deployment is performed without problems.

enter image description here

  1. Pulled the docker image in Kubernetes cluster VM, so it looks like the problem is in the GitOps controller:

enter image description here

  1. To partially solve the problem, the tag has been merged with the custom part of the path. So if before the endpoint was registry.xxx:443/xxx/xxx/xxx/xxx/db:c549f2b6, now it is registry.xxx:443/xxx/xxx/xxx/xxx/xx:db.c549f2b6. This solution is partial, so I leave the issue open.

enter image description here

Question

Any suggestions other than separating the subprojects (sub_proj_1/, sub_proj_2/, sub_proj_n/) into separate repositories, achieving proper authentication of the registry in ArgoCD?

Solution

  • The problem has been solved after trying to answer the following questions:

    Could it be URL address resolving the issue?

    I don't think so, because as mentioned in point 3 of the troubleshooting section. I have removed the custom part of the registry path and now it works properly. Using the same naming convention of GitLab documentation:

    • Failure: registry.example.com/mynamespace/myproject/image:latest
    • Works: registry.example.com/mynamespace/myproject:some-tag

    Could it be an authentication problem?

    I want to think that this is because as mentioned in point 2 of the troubleshooting section, I can manually download the docker image from the cluster.

    image Could the failed deployment manifest be copied and applied to another namespace to check that is not a problem with cluster configuration?

    As I am in the troubleshooting phase the way to test is via argocd from the VM where I have the GitOps Controller:

    argocd app create staging --repo https://gitlab.xxx/xxx.git --path charts/db --dest-server https://X.X.X.X:16443 --dest-namespace <name> --sync-policy automatic --auto-prune --self-heal

    After creating a new namespace in the same cluster the authentication works properly.