Search code examples
rubymultithreadingopen-urifibers

How to Process Items in an Array in Parallel using Ruby (and open-uri)

I am wondering how i can go about opening multiple concurrent connections using open-uri? i THINK I need to use threading or fibers some how but i'm not sure.

Example code:

def get_doc(url)
  begin
    Nokogiri::HTML(open(url).read)
  rescue Exception => ex
    puts "Failed at #{Time.now}"
    puts "Error: #{ex}"
  end
end

array_of_urls_to_process = [......]

# How can I iterate over items in the array in parallel (instead of one at a time?)
array_of_urls_to_process.each do |url|
  x = get_doc(url)
  do_something(x)
end
Solution

  • There's also a gem called Parallel which is similar to Peach, but is actively updated.