How to download a file from GitHub Container Registry (CLI command, GitHub Packages)

Question

How can I download a package from GitHub Packages using the command-line?

I need to download a file from GitHub Packages so that it can be transferred to a machine without an Internet connection for an offline install. But it's not clear how I can download the file.

For example, let's consider the wget package in the Homebrew organization on GitHub Packages.

• https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fwget

I was successfully able to download the manifest with the following command

curl -o manifest.json -v -H "Authorization: Bearer QQ==" -H 'Accept: application/vnd.oci.image.index.v1+json' https://ghcr.io/v2/homebrew/core/wget/manifests/1.24.5

And here is an example execution of the above commands:

user@disp897:~$ curl -o manifest.json -v -H "Authorization: Bearer QQ==" -H 'Accept: application/vnd.oci.image.index.v1+json' https://ghcr.io/v2/homebrew/core/wget/manifests/1.24.5
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.82.121.33:443...
* Connected to ghcr.io (140.82.121.33) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3256 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.ghcr.io
*  start date: Jul 10 00:00:00 2023 GMT
*  expire date: Jul  9 23:59:59 2024 GMT
*  subjectAltName: host "ghcr.io" matched cert's "ghcr.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5f5c1971a810)
} [5 bytes data]
> GET /v2/homebrew/core/wget/manifests/1.24.5 HTTP/2
> Host: ghcr.io
> user-agent: curl/7.74.0
> authorization: Bearer QQ==
> accept: application/vnd.oci.image.index.v1+json
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0< HTTP/2 200 
< content-length: 12448
< content-type: application/vnd.oci.image.index.v1+json
< docker-content-digest: sha256:f7dd153445ffd9ec96cc95d8829cd4afca6ca04b20acbd52cbbd13cfe9aeda5b
< docker-distribution-api-version: registry/2.0
< etag: "sha256:f7dd153445ffd9ec96cc95d8829cd4afca6ca04b20acbd52cbbd13cfe9aeda5b"
< date: Fri, 15 Mar 2024 05:16:30 GMT
< x-github-request-id: B984:0DCC:46A7360:4866BBC:65F3D9AD
< 
{ [1000 bytes data]
100 12448  100 12448    0     0   8491      0  0:00:01  0:00:01 --:--:--  8485
* Connection #0 to host ghcr.io left intact
user@disp897:~$ 

user@disp897:~$ head manifest.json 
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:c5ae04188725dc26a627b5a6309b2c722cff1d1e01ca2dc822bfa0ef5d4bb2e7",
      "size": 2542,
      "platform": {
        "architecture": "arm64",
        "os": "darwin",
user@disp897:~$ 

What I can't figure out, however, is how to parse the manifest file and use it to download the actual package from the GitHub Container Registry.

How can I download an actual package from a GitHub Container Registry using curl?

Answer 1

In 2021, the Homebrew project's free hosing provider (JFrog's Bintray) shutdown. Brew was already hosting their code on GitHub, so I guess someone looked at "GitHub Packages" and figured it was a good (read: free) replacement.

Unfortunately, you can't simply download a file from an OCI Contianer Registry like GitHub Packages. You must:

1. Generate an authentication token for the API
2. Make an API call to the registry, requesting to download a JSON "Manifest"
3. Parse the JSON Manifest to figure out the hash of the file that you want
4. Determine the download URL from the hash
5. Download the file with curl

What is GitHub Packages?

GitHub Packages was launched as a Beta in May 2019. It allowed users to publish packages in many formats, including images uploaded to a GitHub Docker Registry. In September 2020, GitHub added a generic Container Registry as a Beta to GitHub Packages. In June 2021, GitHub migrated all images uploaded to GitHub Packages' Docker Registry (at the domain 'docker.pkg.github.com') to their Container Registry (at the domain 'ghcr.io').

This GitHub Packages Container Registry lets users (like Brew) upload (and download) images in accordance with the Open Container Initiative (OCI) Specifications (described above).

Example

Let's say that we want to download the 'vim' package from the Homebrew project, which is hosted on GitHub Packages.

Determine Package Name

To start, go the org's page on GitHub

https://github.com/Homebrew

Click on "Packages"

Type the name of the package that you want to download

Type the name of the package that you're trying to download into the search bar	The namespace of our 'vim' package is 'homebrew/core/vim'

This will tell you the name of the package, which you may want to cross-check with formulae.brew.sh

The listing for the brew 'vim' package on the GitHub Packages website	The listing for the brew 'vim' package on the Brew website

Determine Package Version

To figure out which version (tag) we want to download from GitHub Packages, we need to check the Brew "recipe". All the brew recipes are found in the Homebrew/homebrew-core repo.

Click the "Formulas" directory, and then click the "v" directory (which holds all the brew formula files for packages that start with the letter 'v').

To view the formulas for all brew packages, go to the 'homebrew-core' repo, and open the "Formula/" directory.	Click the 'v' directory to view all the formulas for packages that start with the letter 'v'

Finally, click the 'vim.rb' file.

You may just want to download the latest version of vim, but let's say that we're using an old machine that can't be updated beyond MacOS 11.7.10 (Big Sur). If you check the latest recipe, you see entries for sonoma (macOS 12), ventura (macOS 13), and monterey (macOS 14). But big_sur is absent because it isn't supported.

The easiest way that I'm aware-of to determine what is the latest brew package's version that supports your OS (without having to fight with the API) is to view the history of the recipe file.

To find the most-recent version to support our OS version, click the "history" icon to view the history of the brew recipe	History of the 'vim.rb' formula (for the 'vim' brew package)

After clicking-through all the previous versions, we can see that big_sur was removed (and not replaced) in a commit on Sep 28, 2023 in commit a153795, which has a commit message "vim: update 9.0.1900_1 bottle". Therefore, it looks like "big_sur" was no longer supported the vim package version '9.0.1900_1'. The version immediately before that is '9.0.1900', and it looks like it does have a package for "big_sur"

The diff shows most of the hashes being updated, but the "big_sur" line was removed (and not replaced)	The most-recent version that supports 'big_sur' is the most-recent version of the 'vim.rb' file that contains a sha256sum for "big_sur"

Now that we know that we want to download '9.0.1900' version of the 'vim' package, we can go back to the GitHub Packages page and find the tag for this version.

Click on "View all tagged versions" and then scroll-down to the tag that corresponds to the version we want (9.0.1900).

To view a list of all of the versions for the package, click "View all tagged versions"	Scroll down to find the tag for the version that you want. In our case, it's "9.0.1900"

Get an Auth Token

Execute the following command to get an authentication token from GitHub Packages.

# get a JSON with an anonymous token
curl -so "token.json" "https://ghcr.io/token?service=ghcr.io&scope=<resourcetype>:<component>/<component>/<component>:<action>";

# extract token from JSON
token=$(cat token.json | jq -jr ".token")

The above commands will get a free/temporary token that you can use in subsequent API calls. If all went well, there will be no output from these commands. Here's an example execution

user@disp7456:~$ curl -so "token.json" "https://ghcr.io/token?service=ghcr.io&scope=repository:homebrew/core/go:pull";
user@disp7456:~$

user@disp7456:~$ token=$(cat token.json | jq -jr ".token")
user@disp7456:~$ 

List the Tags

We can list all of the available tags for the 'vim' package with the 'GET /v2/<name>/tags/list' API endpoint, as shown in the table above

curl -i -H "Authorization: Bearer $" https://ghcr.io/v2/homebrew/core/<package_name>/tags/list

Here's an example execution. Note that it affirms the existence of the '9.0.1900' tag that we found above.

user@disp7456:~$ curl -i -H "Authorization: Bearer $" https://ghcr.io/v2/homebrew/core/vim/tags/list
HTTP/2 200 
content-type: application/json
docker-distribution-api-version: registry/2.0
link: </v2/homebrew/core/vim/tags/list?last=9.0.1500&n=0>; rel="next"
date: Mon, 06 May 2024 01:06:25 GMT
content-length: 1164
x-github-request-id: CD4E:29D249:57F4C1F:5A244F1:66382D11


user@disp7456:~$ 

Download the Manifest

We can download the manifest for the '9.0.1900' tag of the 'vim' package with the 'GET /v2/<name>/manifests/<reference>' API endpoint, as shown in the table above

curl -o manifest.json -s -H "Authorization: Bearer $" -H 'Accept: application/vnd.oci.image.index.v1+json' https://ghcr.io/v2/homebrew/core/<package_name>/manifests/<tag>

And here's an example execution that downloads the manifest for the '9.0.1900' tag of the 'vim' package

user@disp7456:~$ curl -o manifest.json -s -H "Authorization: Bearer $" -H 'Accept: application/vnd.oci.image.index.v1+json' https://ghcr.io/v2/homebrew/core/vim/manifests/9.0.1900
user@disp7456:~$ 

user@disp7456:~$ ls
manifest.json
user@disp7456:~$ 

⚠ Note that we MUST specify the Accept header here. If we don't, then the registry will respond with the following error message

``

For more information on the OCI media types and their MIME types, see the list of OCI Image Media Types.

Parse the Manifest

The previous step downloaded a file named 'manifest.json'. This 'manifest.json' file lists all of the many files available for the '9.0.1900' version of the 'vim' package -- spanning many OS versions and processor architectures.

In order to proceed and download our actual brew bottle file, we need to extract the 'sh.brew.bottle.digest' for our target system, which is a sha256sum hash that we'll use in the 'GET /v2/<name>/blobs/' API call to actually download the file from the container registry.

There's only one dictionary in the 'manifests' array that has a 'platform' dict with the following keys:

1. "architecture": "amd64", and
2. "os.version": "macOS 11.7"

...And that's the one we want:

{
  "schemaVersion": 2,
  "manifests": [
    ...
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:5b538ff92ab00c3658b152dee240e30f9ffa65d817540b6a461460b02b93ceda",
      "size": 5357,
      "platform": {
        "architecture": "amd64",
        "os": "darwin",
        "os.version": "macOS 11.7"
      },
      "annotations": {
        "org.opencontainers.image.ref.name": "9.0.1900.big_sur",
        "sh.brew.bottle.digest": "6cbad503034158806227128743d2acc08773c90890cea12efee25c4a53399d02",
        "sh.brew.bottle.size": "13598246",
        "sh.brew.tab": ""
      }
    },
    ...
  ],
  "annotations": {
    "com.github.package.type": "homebrew_bottle",
    "org.opencontainers.image.created": "2023-09-16",
    "org.opencontainers.image.description": "Vi 'workalike' with many additional features",
    "org.opencontainers.image.documentation": "https://formulae.brew.sh/formula/vim",
    "org.opencontainers.image.license": "Vim",
    "org.opencontainers.image.ref.name": "9.0.1900",
    "org.opencontainers.image.revision": "a78712b62dd7fba03243de22c20e4858d0fd1802",
    "org.opencontainers.image.source": "https://github.com/homebrew/homebrew-core/blob/a78712b62dd7fba03243de22c20e4858d0fd1802/Formula/v/vim.rb",
    "org.opencontainers.image.title": "vim",
    "org.opencontainers.image.url": "https://www.vim.org/",
    "org.opencontainers.image.vendor": "homebrew",
    "org.opencontainers.image.version": "9.0.1900"
  }
}

And, voilà, the block above shows us the 'sh.brew.bottle.digest' that we need in the next step

You can either pick through the file in your fav editor of-choice, but I found (when downloading many files from GitHub Packages) it was much easier to use 'jq'

cat manifest.json | jq '.manifests[] | select(.platform."os.version" | startswith("macOS <version_num>")) | select(.platform.architecture=="<arch>")' | jq -r '.annotations."sh.brew.bottle.digest"'

For example, here's an execution that uses 'jq' to extract the bottle digest hash for the blob for macOS 11 on a system with an amd64 processor.

user@disp7456:~$ cat manifest.json | jq '.manifests[] | select(.platform."os.version" | startswith("macOS 11")) | select(.platform.architecture=="amd64")' | jq -r '.annotations."sh.brew.bottle.digest"'
6cbad503034158806227128743d2acc08773c90890cea12efee25c4a53399d02
user@disp7456:~$ 

Download the file

Now, we can finally download our file if we just pass the 'sh.brew.bottle.digest' hash found above into the GET /v2/<name>/blobs/' API endpoint, as shown in the table above

curl -Lo <package_name>-<package_version>.bottle.tar.gz -H "Authorization: Bearer $" -H 'Accept: application/vnd.oci.image.layer.v1.tar+gzip' https://ghcr.io/v2/homebrew/core/<package_name>/blobs/sha256:<sh.brew.bottle.digest>

For example, the following command downloads 'vim-9.0.1900.bottle.tar.gz'

user@disp7456:~$ curl -Lo vim-9.0.1900.bottle.tar.gz -H "Authorization: Bearer $" -H 'Accept: application/vnd.oci.image.layer.v1.tar+gzip' https://ghcr.io/v2/homebrew/core/vim/blobs/sha256:6cbad503034158806227128743d2acc08773c90890cea12efee25c4a53399d02
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 12.9M  100 12.9M    0     0  1450k      0  0:00:09  0:00:09 --:--:-- 2798k
user@disp7456:~$ 

user@disp7456:~$ ls
 vim-9.0.1900.bottle.tar.gz
user@disp7456:~$ 

Install the file

Finally, you can now transfer the file 'vim-9.0.1900.bottle.tar.gz' to your macOS system and install it with brew.

brew reinstall --verbose --debug path/to/<package_name>-<package_version>.tar.gz

Here's an example execution

user@host ~ % /usr/local/bin/brew reinstall --debug --verbose build/deps/vim-9.0.1900.bottle.tar.gz
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromNameLoader): loading vim
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromBottleLoader): loading build/deps/vim-9.0.1900.bottle.tar.gz
...
␛[34m==>␛[0m ␛[1mSummary␛[0m
🍺  /usr/local/Cellar/vim/9.0.1900: 2,220 files, 40.3MB
...
user@host ~ % 

The package 'vim' is now installed.

Attribution

For more details describing how to download a Brew bottle from GitHub packages, see Manually Downloading Container Images (Docker, Github Packages), from which the examples above were copied.