How to use ruby fibers to avoid blocking IO

I need to upload a bunch of files in a directory to S3. Since more than 90% of the time required to upload is spent waiting for the http request to finish, I want to execute several of them at once somehow.

Can Fibers help me with this at all? They are described as a way to solve this sort of problem, but I can't think of any way I can do any work while an http call blocks.

Any way I can solve this problem without threads?

Solution

I'm not up on fibers in 1.9, but regular Threads from 1.8.6 can solve this problem. Try using a Queue http://ruby-doc.org/stdlib/libdoc/thread/rdoc/classes/Queue.html

Looking at the example in the documentation, your consumer is the part that does the upload. It 'consumes' a URL and a file, and uploads the data. The producer is the part of your program that keeps working and finds new files to upload.

If you want to upload multiple files at once, simply launch a new Thread for each file:

t = Thread.new do
  upload_file(param1, param2)
end
@all_threads << t

Then, later on in your 'producer' code (which, remember, doesn't have to be in its own Thread, it could be the main program):

@all_threads.each do |t|
  t.join if t.alive?
end

The Queue can either be a @member_variable or a $global.