User.all takes 5-10 minutes locally. Request times out to iterate through each

Question

I have 233_997 records in the database for Users. It takes over 10 minutes to iterate through this and I need to go through each one to send a job to sidekiq. The server times out before it can create a job for each user. What do I do?

User.find_each do |user|
  UpdateMembershipLevelWorker.perform_async(user.id)
end

How can I make this request go through without timing out the server? It's the User.find_each part that is very slow... but maybe that's normal with 200k records.

Answer 1

I think what you need to do is , have a background process (via sidekiq , since you are already using it) for your users loop.

# app/workers/hard_worker.rb
class HardWorker
  include Sidekiq::Worker

  User.find_each do |user|
   UpdateMembershipLevelWorker.perform_async(user.id)
  end 

end

(got copied from the site :) http://mperham.github.com/sidekiq/)

What I normally do in long running processes are, run them in background and writing the status to a db table. So later I could display users about the errors etc..