BLOB_UNKNOWN on valid mcr.microsoft.com/windows/servercore:ltsc2019-amd64 container image

Question

For the past few days, I’ve been playing around with the docker registry API and writing a small tool that scopes down its interaction with the Microsoft container registry (mcr.microsoft.com). My ultimate goal is to be able to download an image from MCR without having a direct dependency on docker pull nor any docker tool whatsoever.

Reading through the documentation of the Docker Registry API, specifically the section on how to pull a layer, it states that the URL is built as /v2/<name>/blobs/<digest>. Then, it mentions how clients should be prepared to get a redirect response from such URL.

I’ve been trying to pull image mcr.microsoft.com/windows/servercore:ltsc2019-amd64, but I don’t seem able to achieve this successfully.

From docker, this seems to be working fine:

PS C:\> docker pull mcr.microsoft.com/windows/servercore:ltsc2019-amd64                                                                        ltsc2019-amd64: Pulling from windows/servercore
65014b3c3121: Pull complete                                                                                                                    b16cfeeaf4b3: Pull complete                                                                                                                    Digest: sha256:481b0eb967cee61ce09dd81ece5effc5c327c170d11cc73c307c88a80017c9eb
Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:ltsc2019-amd64
mcr.microsoft.com/windows/servercore:ltsc2019-amd64

However, I’m unable to get to the individual blobs for this image using the docker registry API directly:

PS C:\> (Invoke-RestMethod -Method Get -Uri "https://mcr.microsoft.com/v2/windows/servercore/manifests/ltsc2019-amd64").fsLayers                                                                                                                
blobSum
-------
sha256:b16cfeeaf4b37af9fc146f7043ceb629c1bc50ace967227817e50e47f4a71529
sha256:65014b3c312172f10bd6701a063f9b5aaf9a916c2d2cb843d406a6f77ded3f8d


PS C:\> Invoke-RestMethod -Method Get -Uri "https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:b16cfeeaf4b37af9fc146f7043ceb629c1bc50ace967227817e50e47f4a71529"                                                                      Invoke-RestMethod : {"errors":[{"code":"BLOB_UNKNOWN","message":"blob unknown to
registry","detail":"sha256:b16cfeeaf4b37af9fc146f7043ceb629c1bc50ace967227817e50e47f4a71529"}]}
At line:1 char:1
+ Invoke-RestMethod -Method Get -Uri "https://mcr.microsoft.com/v2/wind ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
PS C:\> Invoke-RestMethod -Method Get -Uri "https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:65014b3c312172f10bd6701a063f9b5aaf9a916c2d2cb843d406a6f77ded3f8d"                                                                      Invoke-RestMethod : {"errors":[{"code":"BLOB_UNKNOWN","message":"blob unknown to
registry","detail":"sha256:65014b3c312172f10bd6701a063f9b5aaf9a916c2d2cb843d406a6f77ded3f8d"}]}
At line:1 char:1
+ Invoke-RestMethod -Method Get -Uri "https://mcr.microsoft.com/v2/wind ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

The returned error seems to be a "not found" rather than a "redirect". How is docker pull figuring out the right link from where to download the layers?

I tried reading through the docker distribution codebase, but I can’t seem to piece together the puzzle. From https://github.com/docker/distribution/blob/master/registry/storage/paths.go, there is some mention on the storage for blobs, which I believe is from where I layers download path are constructed. However, I don’t fully understand how it’s figuring out the real path since it just tries a few of them until one is valid.

What could possibly be wrong here? Am I doing something wrong? Am I missing something?

Answer 1

If you check the docker manifest spec, it says about the foreign layer: https://docs.docker.com/registry/spec/manifest-v2-2/

Layers of type application/vnd.docker.image.rootfs.foreign.diff.tar.gzip may be pulled from a remote location but they should never be pushed.

This mostly applies to Windows base layers which are normally hosted separatly from outside of registry. This is currently the same for MCR. If you look at the manifest of the image, you can see layers with URLs. When server returns 404, you should follow the URLs in the manifest to download the layer blob

   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
         "size": 1534685324,
         "digest": "sha256:65014b3c312172f10bd6701a063f9b5aaf9a916c2d2cb843d406a6f77ded3f8d",
         "urls": [
            "https://go.microsoft.com/fwlink/?linkid=2041275"
         ]
      }