My Rails application generates a ~10MB response dynamically like so:
self.status_code = 200
self.response-body = Enumerator.new { |y| ... }
Starting the application in development mode via unicorn_rails I can curl http://127.0.0.1:3000/foo > /dev/null and see the response being streamed to the client via the progress feedback. When using rails server (which starts webrick), no streaming takes place and I see the progress sitting there, then jumping from 0% to 100%.
Yet when deploying on staging/production, unicorn sits behind nginx and a unix socket. IN that configuration, the similar curl http://staging/foo > /dev/null sees the progress sitting at 0% as the data gets generated, then from 0% to 100% as it gets transferred, just as if there was no streaming. Indeed by looking at the server logs aside I can see that curl sits waiting the whole time the request is generated, and as soon as the request completes, curl trickles up as it receives the data.
My supposition is that something (nginx or whatever) buffers the whole request somewhere before actually sending it to the client.
For now I have played with proxy_buffering off in the nginx configuration, and the X-Accel-Buffering HTTP response header to no avail. I have also bumped nginx 0.7 of Ubuntu 10.04 to 1.2 from the semi-official ppa.
It turned out 10738118 was at the root of the behavior. Indeed, absence of Last-Modified in the response triggers buffering somewhere in Rails/Rack.