Search code examples
nginxdockerblobmanifestnexus

docker pull manifest unknown blob errors


I am currently trying to set up nexus as a private registry for docker images and have been able to login,push,pull,search through the nexus repository .

Currently we are stuck with pulling images from docker if not available within the nexus hosted repository . We are facing errors ranging from blob unknown to manifest unknown.

[root@server1446 ~]$ docker pull server908.int.org.com:6666/centos
Using default tag: latest
Trying to pull repository server908.int.org.com:6666/centos ...
manifest unknown: manifest unknown

Tried following reference [Setup-Docker-Private-Registry-in-Nexus-Repository-OSS-3.0.0][1]

[1]: https://github.com/TerrenceMiao/nexus/wiki/Setup-Docker-Private-Registry-in-Nexus-Repository-OSS-3.0.0 to set up the docker(proxy) and docker(hosted) repo with docker(group) but it throws blob unknown to registry error.

docker(hosted) is configured with http port 4444 & docker(group) with http port 5555 and we have used the same within nginx configuration as below but neither seems to be working .

server {

    listen 6666;
    server_name server908.int.org.com;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/orgnexus.crt;
    ssl_certificate_key /etc/ssl/certs/orgnexus.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    client_max_body_size 1G;
    chunked_transfer_encoding on;

   location / {

      access_log              /var/log/nginx/docker.log;
      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto "https";
      proxy_pass              http://server908.int.org.com:5555;
      proxy_read_timeout      90;

    }

We have commented below entries in "/etc/sysconfig/docker" file .

http_proxy=http://x.x.x.x:3128 https_proxy=http://x.x.x.x:3128


Solution

  • Below are my configurations that got it working .

    server {
    
        proxy_send_timeout 120;
        proxy_read_timeout 300;
        proxy_buffering    off;
        tcp_nodelay        on;
    
        server_tokens off;
        client_max_body_size 1G;
    
        listen 80;
        server_name box.company.net;
        location / {
              rewrite ^(.*) https://box.company.net$1 301;
        }
    }
    
    server {
    
        listen 443;
        server_name box.company.net;
        keepalive_timeout 60;
        ssl on;
        ssl_certificate /etc/ssl/certs/ssl.crt;
        ssl_certificate_key /etc/ssl/certs/ssl.key;
        ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
        ssl_session_cache shared:TLSSSL:16m;
        ssl_session_timeout 10m;
        ssl_prefer_server_ciphers on;
    
        location / {
    
          proxy_set_header        Host $http_host;
          proxy_set_header        X-Real-IP $remote_addr;
          proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header        X-Forwarded-Proto "https";
          proxy_pass              http://box.company.net:8081;
          proxy_read_timeout      90;
    
        }
    }
    
    # correlates to your nexus http connector
    server {
    
        listen 6666;
        server_name box.company.net;
        keepalive_timeout 60;
        ssl on;
        ssl_certificate /etc/ssl/certs/ssl.crt;
        ssl_certificate_key /etc/ssl/certs/ssl.key;
        ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
        ssl_session_cache shared:TLSSSL:16m;
        ssl_session_timeout 10m;
        ssl_prefer_server_ciphers on;
        client_max_body_size 1G;
        chunked_transfer_encoding on;
    
        location / {
    
          proxy_set_header        Host $http_host;
          proxy_set_header        X-Real-IP $remote_addr;
          proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    
            if ($request_method !~* GET) {
                    proxy_pass              http://box.company.net:4444;
            }
            if ($request_method = GET) {
                    proxy_pass              http://box.company.net:5555;
            }
          proxy_read_timeout      90;
        }
    }
    

    Commented below entries with "/etc/default/docker" file .

    http_proxy=http://x.x.x.x:3128
    https_proxy=http://x.x.x.x:3128
    

    Restart Nginx.

    Perform Login

    [test@server ~]$ docker login -u admin -p admin123 box.company.net:6666
    Login Succeeded
    

    Post login a file name "config.json" will be created under ".docker" directory

    [test@server ~]$ cat ~/.docker/config.json
    {
            "auths": {
                    "box.company.net:6666": {
                            "auth": "YWRtaW46YWRtaW4xMjM="
                    }
            }
    }
    

    Search for Images available in docker hub.

    [test@server ~]$ docker search box.company.net:6666/ubuntu
    INDEX      NAME                                                                               DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
    company.net   box.company.net:6666/ubuntu                                                 Ubuntu is a Debian-based Linux operating s...   6186      [OK] 
    

    Pull the images from docker hub via nexus proxy.

    [test@server ~]$ docker pull box.company.net:6666/ubuntu
    Using default tag: latest
    Trying to pull repository box.company.net:6666/ubuntu ...
    sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4: Pulling from box.company.net:6666/ubuntu
    
    75c416ea735c: Pull complete
    c6ff40b6d658: Pull complete
    a7050fc1f338: Pull complete
    f0ffb5cf6ba9: Pull complete
    be232718519c: Pull complete
    Digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4
    Status: Downloaded newer image for box.company.net:6666/ubuntu:latest
    

    Tag The pulled images

    docker tag box.company.net:6666/ubuntu:latest box.company.net:6666/ubuntu:1
    

    Push to NexusHostedRepo (Port : 4444)

    [test@server ~]$ docker push box.company.net:6666/ubuntu:1
    The push refers to a repository [box.company.net:6666/ubuntu]
    0566c118947e: Pushed
    6f9cf951edf5: Pushed
    182d2a55830d: Pushed
    latest: digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4 size: 1357
    

    Pull from Nexus Repo (This should be quick than the pull from docker hub )

    [test@server ~]$ docker pull box.company.net:6666/ubuntu:1
    Trying to pull repository box.company.net:6666/ubuntu ...
    sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4: Pulling from server908.int.org.com:6666/ubuntu
    
    75c416ea735c: Pull complete
    c6ff40b6d658: Pull complete
    a7050fc1f338: Pull complete
    Digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4
    Status: Downloaded newer image for box.company.net:6666/ubuntu:1
    

    --------------------------------------------------------------------------------

    Also make sure you add you proxy server certificate within Nexus SSL Certificate Section .

    keytool -J-Dhttps.proxyHost=<proxy_hostname> -J-Dhttps.proxyPort=<proxy_port> -printcert -rfc -sslserver <remote_host_name:remote_ssl_port>
    

    Replace proxy_hostname and proxy_port with the HTTP proxy server that Nexus is configured with under Administration -> Server. Replace remote_host_name:remote_ssl_port with one of the remote host and port having the certification problem. You can omit the port if it is the default 443. For docker it will be registry-1.docker.io:443

    You should see at least two entries printed by the above command. Take the last certificate content printed and copy it entirely to your clipboard. This should be your proxy server's certificate, added to the end of the certificate chain.

    The copied certificate content should start with -----BEGIN CERTIFICATE----- and end with-----END CERTIFICATE-----.

    Then in the Nexus UI, go to Administration -> SSL Certificates and click Add... and choose Paste PEM. Paste the certificate contents to the dialog that opens.

    Click Load Certificate. Verify the certificate contents in the next window. Verify the Issuer details listed are from your proxy server certificate. When you are satisfied click Add Certificate.


    Hope this helps.