Search code examples
dockerblobmanifestnexusdocker-image

Nexus Repository: BLOB_UNKNOWN: blob unknown to registry

  • The developers regularly worked with the group docker repository. And one day a complaint was received that push images were not possible.
  • When building a container, a group repository is used, which includes: docker-hosted(local), hub.docker(proxy), gcr.io(proxy), mcr.microsoft.com(proxy).
  • Logs during push:
2024-01-16 13:27:51,999+0000 ERROR [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:0ec0ce3058439b66991147caf6fd33b737d8f06dec9c07885ecf836cae3d2cdd for: project/name/1.0.37404 in repository RepositoryImpl$$EnhancerByGuice$$698854289{type=hosted, format=docker, name='project-docker-hosted'}
2024-01-16 13:27:52,012+0000 ERROR [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:c78ba7738ca3a1b0a85ee1fab5a5e8dbfcef6fdb5e515d748e168ce7e804a542 for: project/name/1.0.37404 in repository RepositoryImpl$$EnhancerByGuice$$698854289{type=hosted, format=docker, name='project-docker-hosted'}
2024-01-16 13:27:52,015+0000 ERROR [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:7a3506e858bfcef5d2ea86c950664bdc6cf6c5ef6f278abd2d31f3285ea1e5d2 for: project/name/1.0.37404 in repository RepositoryImpl$$EnhancerByGuice$$698854289{type=hosted, format=docker, name='project-docker-hosted'}
2024-01-16 13:27:52,028+0000 WARN  [qtp69617637-29347] user-git org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/project/name/manifests/1.0.37404: 400 - org.sonatype.nexus.repository.docker.internal.V2Exception: Invalid Manifest

I checked the problematic blobs. And they are indeed missing from the docker-hosted repository, however I was able to find them in mcr.microsoft.com(proxy). And I can't understand why when pushing the problematic blobs are not sent to the hosted-repository...

Push is initially performed by gitlab-runner. I tried manual push, but it didn't help. I also tried rebuild index and invalidate cache for docker-hosted and docker-group repositories respectively. I tried cleaning the nexus using tasks. I also tried downloading the nginx:latest image, changing the tag and uploading it to the docker-hosted repository. But I get an error:

The push refers to repository [project-docker-hosted/nginx]
009507b85609: Layer already exists 
fbcc9bc44d3e: Layer already exists 
b4ad47845036: Layer already exists 
eddcd06e5ef9: Layer already exists 
b61d4b2cd2da: Layer already exists 
b6c2a8d6f0ac: Layer already exists 
571ade696b26: Layer already exists 
errors:
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
blob unknown: blob unknown to registry
Solution

  • I was using nginx as a reverse proxy and the problem was the distribution of requests. In my original implementation of the config there was only one location ~ ^/(v1|v2)/ and redirection took place in it depending on the type:

    map $request_method $redirection {
    default                    "nexus";
    "~(GET|HEAD)"              "docker-group";
    "~(PATCH|PUT|POST|DELETE)" "docker-hosted";
}

    However, in this case, requests of type HEAD are ALWAYS redirected to the group repository, which is why the error occurred when pushing. Because blob was present in the group repository, but not in the local one.

    To solve the problem, I added another map and location.

    map:

    map $request_method $redirection {
    default                    "nexus";
    "~(GET|HEAD)"              "docker-group";
    "~(PATCH|PUT|POST|DELETE)" "docker-hosted";
}

map $request_method $redirection-blobs {
    default                         "nexus";
    "~(GET)"                        "docker-group";
    "~(PATCH|PUT|POST|DELETE|HEAD)" "docker-hosted";
}

    location:

       location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
     proxy_set_header Host $host:$server_port;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto "https";

     proxy_pass http://$redirection-blobs;
    }

   location ~ ^/(v1|v2)/ {
     proxy_set_header Host $host:$server_port;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto "https";

     proxy_pass http://$redirection-main;
    }

    After which he pushed successfully!