Search code examples
dockerartifactory

Convert artifactory blobs to docker registry structure

Could anyone point me to description of directory structure that docker registry relies on?

Background

I cannot pull docker image from our company's artifactory server. All my invocations of docker pull art.server.corp/repo/image:label end with error Unexpected EOF. My colleagues report the same issue.

I have asked for help to our Artifactory support, but waiting for their response takes time, and I don't believe in their result.

In the meanwhile I was able to download contents of that image from a browser through a www-interface of the Artifactory.

I've downloaded file manifest.json and bunch of files with names sha256__<long string with hash>. Most of them are .tar.gz archives and one is in the JSON format.

How can I import these files into my local docker installation? My goal is to have the same container image as in the registry.

I am new to Docker.

I've tried docker load and docker import. The result is not that expected.

docker load complains about missing json files and does nothing.

$ for f in sha256* ; do docker load < $f  ; done
open /var/lib/docker/tmp/docker-import-234007886/var/json: no such file or directory
...
open /var/lib/docker/tmp/docker-import-777861766/var/json: no such file or directory
$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

docker import creates a separate image for each file, while they seem to be different layers of a single file system.

$ for f in sha256* ; do docker import $f image:label; done
sha256:a19634d70ff568616b9c42a0151ae8abbd6b915cb24e9d127e366e14453a0dd4
sha256:28c559e39d3be8267757ba8ca540c6b8440f66b71d4ed640fff1b42a04aa54c5
...
sha256:...

$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
image        label     7a4c9fe6210c   About a minute ago   111MB
<none>       <none>    654768c91f55   About a minute ago   404kB
<none>       <none>    85d37d403e34   About a minute ago   1.37GB
<none>       <none>    28c559e39d3b   About a minute ago   63.2MB

Update

I have tried installing Artifactory-OSS with hope to import these blobs in it, but it seems, Docker is not supported in OSS.

Then I've decided to launch local docker registry and import them in it.

I have developed the following script that copies files to blobs and creates files named link.

#!/bin/zsh

source_dir=/path/to/downloaded/blobs
registrty_root=/mnt/volume/docker/registry/v2
image_name=image
image_tag=label


blobs=$registrty_root/blobs/sha256
repo_path=$registrty_root/repositories/$image_name
layers=$repo_path/_layers/sha256
manifests=$repo_path/_manifests/revisions/sha256

#debug=echo
debug=

for f in $source_dir/sha256* ; do
    echo $f
    bn=$(basename $f)
    sha256=$(echo $bn | cut -c9-)
    first2=$(echo $sha256 | cut -c-2)
    blob_dir=$blobs/$first2/$sha256
    layer_dir=$layers/sha256/$sha256
    $debug mkdir -p $blob_dir
    $debug cp -v $f $blob_dir/data

    if [[ $(file $f)=~"gzip" ]] then ; 
        $debug mkdir -p $layer_dir
        $debug echo "sha256:$sha256" > $layer_dir/link
    else
        $debug mkdir -p $manifests
        $debug echo "sha256:$sha256" > $manifests/link
    fi
done

man_sha256=$(sha256sum $source_dir/manifest.json)
first2=$(echo $man_sha256 | cut -c-2)
$debug mkdir -p $blobs/$first2/$man_sha256
$debug cp -v $source_dir/manifest.json $blobs/$first2/$man_sha256/data

$debug mkdir -p $manifests/$man_sha256
$debug echo "sha256:$man_sha256" >  $manifests/$man_sha256/link

$debug mkdir -p $repo_path/_manifests/tags/$image_tag/{current,index/sha256/$man_sha256}
$debug echo "sha256:$man_sha256" > $repo_path/_manifests/tags/$image_tag/index/sha256/$man_sha256/link
$debug echo "sha256:$man_sha256" > $repo_path/_manifests/tags/$image_tag/current/link

Then I launch local docker registry: docker run -p 5000:5000 -v /mnt/volume/docker:/var/lib/registry registry:2

However, when I try to retrieve the image, I get error message:

$ docker pull localhost:5000/image:label

Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

Docker console shows the following:

time="2021-01-18T14:04:54.269748704Z"  level=error msg="response completed with error" err.code=unknown err.detail="invalid checksum digest format" err.message="unknown error" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=7d9fdc93-e74f-42d6-a722-81b51c371df5 http.request.method=HEAD http.request.remoteaddr="172.17.0.1:39770" http.request.uri="/v2/image/manifests/label" http.request.useragent="docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/5.4.0-62-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=21.43068ms http.response.status=500 http.response.written=70 vars.name="image" vars.reference="label"

I've also consulted with this link: https://notsosecure.com/anatomy-of-a-hack-docker-registry/ and tried to request manifest from a browser. Tag list is retrieved successfully, but manifest gives error.

So, it looks like I'm missing something. Any help?

Solution

  • After closer acquaintance with Docker and Dockerfiles, I see that it was possible to rebuild this image from those archives using ADD instruction.

    From https://docs.docker.com/engine/reference/builder/#add

    If '< src >' is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory.