Apparently this is a rather known issue: Vagrant/VirtualBox/Apache2 Strange Cache Behaviour, http://smotko.si/nginx-static-file-problem/, https://twitter.com/meinharrd/status/580098162716774400, that attempting to use VirtualBox with Nginx or Apache exhibits buggy behavior (modifying a file will update the contents, but not the length, so making it shorter leads to garbage at the end, and making it longer just truncates at the original length). The solution is always to disable sendfile.
So I disabled sendfile and this actually fixed my problem, but now I see all of those static files taking 2-3 seconds to load every time.
Has anyone seen this sort of behavior before? I'm specifically using Boot2Docker to run Nginx and an app in PHP
There is a known issue on github about volume performance in boot2docker: https://github.com/boot2docker/boot2docker/issues/593
In that issue there is an interesting link about a A productive development environment with Docker on OS X.