Rake Task timing out on Heroku - Move Rake task to worker using Sidekiq

Question

I recently realized that my rake tasks can time out on Heroku, and I happen to have...a lot of them.

Here is an example of my rake task:

namespace :test do

  desc "Test Rake Task"
  task get_list: :environment do
    require 'net/ftp'
    sftp = Net::SFTP.start('ftp.test.com','luigi', :password => 'pass_word')
    records = sftp.download!("luigi/list.xml")
    records_hash = Hash.from_xml(records)
    records_hash['Report']['Details'].each do |record|
      contact = Contact.create(              
          first_name: record['FirstName'],
          last_name: record['LastName'],
          date_of_birth: record['DateofBirth']
      )
      if contact.valid?
        puts "Created contact"
      else
        puts "Invalid contact"
      end
    end    
  end
end

I think that I need to move this to a background worker. It could take 5+ minutes to loop through this task and add each contact to the database. I would like to use Sidekiq, but I've never used "workers" or set this up before.

In general, how would I go about setting Sidekiq up to run on Heroku, and then move the above task to a background worker? I would like for this "worker" or "task" to be scheduled once a week, Monday mornings at 8 am.

My app settings:

Procfile:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb

I am using a postgres Database, Rails 4.0.0, ruby 2.0.0, and as I mentioned earlier my app is hosted on Heroku.

Answer 1

You should follow the Sidekiq documentation and get it setup for Heroku.

One you have Sidekiq running, I recommend the following architecture:

• ContactListWorker: downloads list, enqueues a contact update job for each
• ContactWorker: given details, create/update a contact
• rake task contacts:nightly_update to queue up ContactListWorker job

Here's some rough pseudo code to show what that architecture might look like:

# rake contact:nightly_sync_list
namespace :contacts do
  desc "Test Rake Task"
  nightly_list_sync: :environment do
    ContactListWorker.perform_async
  end
end

class ContactListWorker
  require 'net/ftp'
  include Sidekiq::Worker

  def perform()
    sftp = Net::SFTP.start('ftp.test.com','luigi', :password => 'pass_word')
    records = sftp.download!("luigi/list.xml")
    records_hash = Hash.from_xml(records)
    records_hash['Report']['Details'].each {|record| ContactWorker.perform_async(record) }
  end
end

class ContactWorker
  include Sidekiq::Worker

  def perform(record)
    contact = Contact.create(
      first_name: record['FirstName'],
      last_name: record['LastName'],
      date_of_birth: record['DateofBirth']
    )
    if contact.valid?
      puts "Created contact"
    else
      puts "Invalid contact"
    end
 end

end

This architecture allows you to asynchronously kick off one background ContactListWorker job. This job will do the download and quickly enqueue N ContactWorker jobs asynchronously. This allows you to fan-out the processing across multiple Sidekiq workers and threads to distribute the processing.