I'm not sure if this is possible with Varnish, but I have a backend server that generates large (1-2GB) files, but is located on a relatively slow connection.
Is it possible to configure Varnish (running on a remote machine with a fast connection) to serve multiple users at the same time, but only opening one connection to the backend?
For example, user 1 starts downloading a file through Varnish from the backend, and partway through, user 2 also begins a download. Rather than opening a new connection, could Varnish serve everything it has already cached (say, 100MB), and then once user 2 "catches up", it keeps downloading at the speed of the backend connection (because Varnish is serving the same content to both users). After that initial period, that file would be fully cached on Varnish and is served fast.
Is this something Varnish could be configured to do (or is there a better proxy/cache software for this use case).
What you're describing is called request coalescing and this is a standard Varnish feature.
As long as the object is stored in cache, the streamed content that is partially stored in cache will be consumed by all clients consuming that resource.
I don't think that opening up a 2nd connection is really a problem, but I guess your thought process is avoiding a cache miss for partially cached data. No worries though, Varnish has you covered.