Avoid bazel-remote-cache collision for different Linux versions

Question

Our company is migrating from a lower version of Ubuntu Linux to a higher version. In this process, some dogfooders are using the higher version, while others are using the lower version. We find that some C++ libraries in bazel may share the same 'key' on bazel remote cache across Ubuntu Linux versions. As a result, the building result of one C++ library on Ubuntu Linux version A may be fetched by Ubuntu Linux version B, even though they use different versions of C++ compilers, libstdc++, and pre-built external libraries. This can cause serious problems.

Is there an elegant way to guarantee the C++ building on different versions of Ubuntu Linux do not share the same key on the bazel remote cache?

Currently, what we did is to let our build tool automatically generate an auxiliary bazelrc file which appends a parameter '--copt=-D__UBUNTU_VERSION_XX_YY'. This C macro is never used in our C++ code. Rather, it just makes sure the C++ building actions in different versions of the Ubuntu Linux do not share the same key on the remote cache. While this works, we think there must be something more elegant.

Answer 1

There's two options:

1. Make the build completely hermetic, so it has minimal dependencies on the system. Besides making the remote cache trustworthy, hermeticity has reproducibility and usability benefits—developers need only Bazel installed and a source tree to build anything and trust they would obtain the same output as anyone else. Of course, expurgating external dependencies from a build can be a long process.
2. Manually modify the cache key as you've done. There's an upstream issue to do system toolchain detection automatically. I wouldn't hope for a magical solution here, though. It's tricky to mechanically balance distinguishing system differences that affect the build and distinguishing every individual client (making the remote cache useless).