The theory that "disk" is cheap has gotten a bit out of hand lately. There are some powerful aspects of version control that have enabled us to onboard new developers with a few bootstrap files and one simple command to pull the toolchain over.
Recently upgrades to the systems have prompted requests for storing built binaries. This has been followed on by a request to version the entire virtualized build system. Each layer added on top creates important relationships between repositories and a good fundamental design is necessary to manage it.
The storing of the toolchain brought instant benefit while the storing of the built binaries brough instant liabilities. Git, unfortunately, has some fundamental issues when dealing with large binary files.
Where do you draw the lines at using VC in the right ways and when do you start investigating more appropriate solutions?
You probably shouldn't be storing the "entire virtualized build system" as a giant binary. If you want to version your application, you version the source code, not the compiled binary.
What many shops do is store in version control the steps to recreate the build server. Then you need one fixed image (a stock, out-of-the-box OS install), plus a small number of files (what to install on it, and how). Some places even have their server rebuild the app from source, on a clean OS install, for every deploy/reboot.
Versioning the OS image itself as a giant binary isn't nearly as useful. You can't branch. You can't merge. You can't diff. What's the point? You might save space if your VCS can do binary diffs, but that probably takes a ton of CPU and memory to do, and if they're on a "disk is cheap" binge, then there's no reason to make life painful just to save disk space.
Either store your install scripts/libraries in VC and rebuild the VM image as needed, or just store VM images in normal files. I don't see any point in putting the images in VCS.