How to download a delayed file via HTTP in Ruby?

Question

I use the following Ruby function to download various files via HTTP:

def http_download(uri, filename)
  bytes_total = nil
  begin
    uri.open(
      read_timeout: 500,
      content_length_proc: lambda { |content_length|
        bytes_total = content_length
      },
      progress_proc: lambda { |bytes_transferred|
        if bytes_total
          print("\r#{bytes_transferred} of #{bytes_total} bytes")
        else
          print("\r#{bytes_transferred} bytes (total size unknown)")
        end
      }
    ) do |file|
      open filename, 'w' do |io|
        file.each_line do |line|
          io.write line
        end
      end
    end
  rescue => e
    puts e
  end
end

I also want to download files (csv, kml, zip, geojson) from this website. However, there is some kind of delay set up. When I click the download link in the browser it takes a bit until the download window appears. I suppose the file needs to be processed on the server before it can be served.

How can I modify my script to take the delay into account?

I am running Ruby 2.2.2.

Answer 1

Here is the modification according to the post and comment:

require 'open-uri'

def http_download(uri, filename)
  bytes_total = nil
  index = 1
  begin
    open(
      uri,
      read_timeout: 500,
      content_length_proc: lambda { |content_length|
        bytes_total = content_length
      },
      progress_proc: lambda { |bytes_transferred|
        if bytes_total
          print("\r#{bytes_transferred} of #{bytes_total} bytes")
        else
          print("\r#{bytes_transferred} bytes (total size unknown)")
        end
      }
    ) do |io|
      # if "application/json" == io.content_type
      if io.is_a? StringIO
        raise " --> Failed, server is processing. Retry the request ##{index}"
      else # Tempfile
        puts "\n--> Succeed, writing to #{filename}"
        File.open(filename, 'w'){|wf| wf.write io.read}
      end
    end
  rescue => e
    puts e
    return if e.is_a? OpenURI::HTTPError # Processing error

    index += 1
    return if index > 10

    sleep index and retry
  end
end