rails with sidekiq, race condition

in my rails (v 4.2.11.3) app, I have some legacy logic that when I create a new model this gets created and then updated a couple of times, with additional data after the same call. Resulting in three separate commits to the database.

I have an after commit callback that is used to kick some asynchronous job that keeps the data aligned to a remote service using sidekiq.

Now it's happening that the sidekiq is enqueued three times and I get a race condition when the jobs start at the same time.

I wrapped the jobs in transitions (on the actual object being worked on, I don't want to wrap the entire model in a transition), but this doesn't seem like working, I byebugged the code and saw separate jobs entering the transition at the same time.

this is the method that is called in the sidekiq job,

 def save_object(id)
   object = Object.find(id)

   object.transaction do
     object.reload

     byebug
     if object.service_id.blank?
       create_object(object)
     else
       update_object(object)
     end
   end
 end

and this is the byebug log showing that two sidekiq jobs are kicked at the same time, and they can both reach the byebug call that is inside the transaction block.

  [23, 29] in /Users/giulio/my-app/app/services/object_saving_service.rb
     23:
     24:       byebug
  => 25:       if object.service_id.blank?
     26:         create_object(object)
     27:       else
     28:         update_object(object)
     29:       end
  (byebug) th current
  + 1 #<Thread:0x00007fad30298dc8@/Users/giulio/.rvm/gems/ruby-2.3.8/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:24 run> /Users/giulio/my-app/app/services/object_saving_service.rb:25
  (byebug) object.service_id
  nil
  (byebug) n

  [23, 29] in /Users/giulio/my-app/app/services/object_saving_service.rb
     23:
     24:       byebug
  => 25:       if object.service_id.blank?
     26:         create_object(object)
     27:       else
     28:         update_object(object)
     29:       end
  (byebug) th current
  + 3 #<Thread:0x00007fad30299ef8@/Users/giulio/.rvm/gems/ruby-2.3.8/gems/sidekiq-4.2.10/lib/sidekiq/util.rb:24 run> /Users/giulio/my-app/app/services/object_saving_service.rb:25
  (byebug) object.service_id
  nil
  (byebug)

the way I expect transaction to work is that only one of these blocks can be executed at any time, so the first one finds: object.service_id.nil? == true and executes the create_object, while the second finds object.service_id.nil? == false and executes an update.

in the log above both times the object.service_id is nil, and both do a create_object call. generating a race condition.

any ideas of why the transition is not working? how could I get it to work?

Thanks.

Solution

DB Transactions are not a lock, they don't guarantee the code won't be accessed simultaneously by multiple threads / processes. It seems to me your example needs a row lock https://api.rubyonrails.org/v6.1.3/classes/ActiveRecord/Locking/Pessimistic.html. So locking (object in your example) could be the way to go.

object.with_lock do
   object.reload

   byebug
   if object.service_id.blank?
     create_object(object)
   else
     update_object(object)
   end
 end
end