Using limit and offset in rails together with updated_at and find_each - will that cause a problem?

Question

I have a Ruby on Rails project in which there are millions of products with different urls. I have a function "test_response" that checks the url and returns either a true or false for the Product attribute marked_as_broken, either way the Product is saved and has its "updated_at"-attribute updated to the current Timestamp.

Since this is a very tedious process I have created a task which in turn starts off 15 tasks, each with a N/15 number of products to check. The first one should check from, for example, the first to the 10.000th, the second one from the 10.000nd to the 20.000nd and so on, using limit and offset.

This script works fine, it starts off 15 process but rather quickly completes one script after another far too early. It does not terminate, it finishes with a "Process exited with status 0".

My guess here is that using find_each together with a search for updated_at as well as in fact updating the "updated_at" while running the script changes everything and does not make the script go through the 10.000 items as supposed but I can't verify this.

Is there something inherently wrong by doing what I do here. For example, does "find_each" run a new sql query once in a while providing completely different results each time, than anticipated? I do expect it to provide the same 10.000 -> 20.000 but just split it up in pieces.

task :big_response_launcher => :environment do
  nbr_of_fps = Product.where(:marked_as_broken => false).where("updated_at < '" + 1.year.ago.to_date.to_s + "'").size.to_i
  nbr_of_processes = 15
  batch_size = ((nbr_of_fps / nbr_of_processes))-2
  heroku = PlatformAPI.connect_oauth(auth_code_provided_elsewhere)  
  (0..nbr_of_processes-1).each do |i|
    puts "Launching #{i.to_s}"
    current_offset = batch_size * i
    puts "rake big_response_tester[#{current_offset},#{batch_size}]"
    heroku.dyno.create('kopa', {
      :command => "rake big_response_tester[#{current_offset},#{batch_size}]",
      :attach => false
    }) 
  end

end

task :big_response_tester, [:current_offset, :batch_size] => :environment do |task,args|
  current_limit = args[:batch_size].to_i
  current_offset = args[:current_offset].to_i  
  puts "Launching with offset #{current_offset.to_s} and limit #{current_limit.to_s}"
  Product.where(:marked_as_broken => false).where("updated_at < '" + 1.year.ago.to_date.to_s + "'").limit(current_limit).offset(current_offset).find_each do |fp|
    fp.test_response
  end  
end

Answer 1

As many have noted in the comments, it seems like using find_each will ignore the order and limit. I found this answer (ActiveRecord find_each combined with limit and order) that seems to be working for me. It's not working 100% but it is a definite improvement. The rest seems to be a memory issue, i.e. I cannot have too many processes running at the same time on Heroku.