Can I display a slow-loading web page incrementally?

Question

I'm working on mashup that scrapes a couple of sites for data. I want to scrape and cache the data on demand rather than index the entire sites.

The first time the data is fetched, the operation can be extremely slow--at least a couple of minutes.

What's the best practice for displaying pages line-by-line like this? Is there a way to display the page dynamically, showing data as it's fetched?

Related:

• How to display HTML to the browser incrementally over a long period of time?
• How to create an Incremental loading webpage
• How to make sure an HTML table renders incrementally
• Display the result on the webpage as soon as the data is available at server

Answer 1

It looks like Rails Live Streaming is one approach. Disadvantage is that it appears to be highly web server dependent and touchy. Here's a tutorial:

http://tenderlovemaking.com/2012/07/30/is-it-live.html

class MyController < ActionController::Base
  include ActionController::Live

  def index
    100.times {
      response.stream.write "hello world\n"
    }
    response.stream.close
  end
end