Server farm and/or other solutions for running a ruby script fast

Question

I have a ruby script that's going to take about 5 hours to crunch on my specced-out rmbp. Problem: We need it within 2 hours.

The code being run is non-threadsafe, and spits out a csv file from a xlsx input. ...I've never used server farms before, but I'm guessing non-threadsafe ruby isn't exactly their thing(?)

In short, is there any sort of server farm or service or any way at all I can crunch a ruby script that takes 5 hours in less than an hour or two?

Answer 1

I've done something similar using Heroku and Sidekiq. Heroku provides both free and mini-plans for low-cost computing, and the Sidekiq gem lets you chunk out your work to several workers, letting them run simultaneously, thus finishing faster.

Do you know where your bottleneck is? Is it the input of the XLSX file or the output to CSV? If it's the latter, bring in your XLSV file, and chunk bits of it out to Sidekiq workers.

For example, you could split out your XLSX file into chunks of 1000 rows, then

rows.each do |row|
  MySidekiqWorker.perform_async(row)
end

And your MySidekiqWorker might look like:

class MySidekiqWorker
  include Sidekiq::Worker

  def perform(row)
    # append processed data to CSV
  end
end

Your workers do all the heavy lifting and will almost certainly be faster than your laptop without such help.