How to improve an ActiveRecord query

Question

I have a method which returns the results from a query. The code that calls the method then loops through each result and launches a sidekiq worker. The problem I'm running into is that the looping is actually taking a decent amount of time (almost the same amount of time it takes to run all the workers). Here's the query:

Object.where("last_updated > ?" , 1.days.ago.midnight )

I then do the following:

objects.each { |o| o.perform_async(something) }

I'm trying to figure out how to make this process more efficient. The result is that it's taking around 10 minutes for this process to complete, effectively taking 20 milliseconds per launch (if the query returns 30,000 results). Is there any way to make this quicker?

Answer 1

I see that you already have last_updated indexed. Next:

Object.select('id, only_columns_you_need').where(...).find_each do |object|
  object.perform_async(something)
end

If the "objects" table has lots of columns but you only need a few for this operation, selecting only those columns can really speed things up in both db and Ruby land.

find_each will load the records in batches of 1000 by default. Use the :batch_size option to tweak that.

UPDATE

def do_stuff_to_objects(&stuff)
  Object.select('id, only_columns_you_need').where(...).find_each(&stuff)
end

...

do_stuff_to_objects do |object|
  object.perform_async(something)
end