Search code examples
nginxbazelhttp-proxy

Dependency archiving for airgapped network


I am trying to use Bazel in a network which is not connected to the internet. Even without using any special libraries, Bazel still needs to download some dependencies via HTTP.

I was advised to try and set up an HTTP proxy server which will cache the files which Bazel downloads, and then transfer the proxy server along with its cache into our isolated network.

In order for this to work, I need to be able to operate the proxy in two modes:

  1. When creating the cache, the proxy caches all requests from Bazel, and never discards any of the files.
  2. When using the cache within the network, the proxy never tries to cache any additional files, serves the existing files in the cache and never discards them.

I looked at nginx in order to achieve this, but could not find any option to prevent files in the cache from being discarded.

Is there any way to do this in nginx? If not is there a proxy which suites my use case? Or maybe there's another alternative solution which I haven't considered?


Solution

  • You can use the --distdir distribution directory in your airgapped environment. This distribution directory should contain the implicit dependencies of the Bazel binary you're invoking.

    To build these dependencies outside of your airgapped environment, first checkout the Bazel source tree at the right version:

    git clone https://github.com/bazelbuild/bazel "$BAZEL_DIR"
    cd "$BAZEL_DIR"
    git checkout "$BAZEL_VERSION"
    

    Then, build the tarball containing the implicit runtime dependencies for that specific Bazel version:

    bazel build @additional_distfiles//:archives.tar
    

    Export this tarball to a directory that can be copied into your airgapped environment. Note the --strip-components flag, because --distdir can be quite finicky with the directory nesting level:

    tar xvf bazel-bin/external/additional_distfiles/archives.tar -C "$NEW_DIRECTORY" --strip-components=3
    

    Finally, when you use Bazel in your airgapped environment, pass the --distdir flag pointing to the directory. I suggest adding it as an .bazelrc entry:

    build --distdir=path/to/directory