Can single delayed job worker start the next job before the ongoing job ends?

Question

I have implemented API rate limiting in my application for calling 3rd party API service which limits the number of requests made to service in a specified time interval.

I went through delayed job documentation and could not find the information i was looking for.

My case is;

If there is a really short amount of time, let's say 5 seconds, between 2 delayed jobs, what will the worker do?

Will it wait for the first job to finish even though the second job execution time has arrived, or will it start the job on the specified time?

If the latter, It will break my rate limiting implementation since API is only letting me 60 requests per minute and and both jobs will just limit themselves to 60 requests per minute and in total both jobs will try to send around 120 requests in a minute.

Thanks in advance, cheers!

Answer 1

This depends on the numbers of DelayedJob workers that are available. Because a free worker will pick a job and process it as soon as its run_at time has come.

If you need to ensure that only on job runs at a time, I see two simple options:

1. limit your number of workers to 1. Or
2. enqueue the next job as the last step of processing.

That might look like this:

class Job
  def process
    # code for the job ... 

    # enqueue the next job to run in 5 seconds
    Job.delay(run_at: 5.seconds.from_now).process
  end
end

A third option that is a bit more complex, plays with named queues. When you enqueue this kind of jobs into a special named queue, then you can use that special queue to calculate the time for the next job. Instead of just enqueue a job like this:

Job.delay.process

Check if a job already exists, like this:

queue_name = 'one_at_a_time'
latest_job = Delayed::Job.where(queue: queue_name).order(:run_at).last
run_at     = latest_job ? latest_job.run_at + 5.seconds : Time.current
Job.delay(queue: queue_name, run_at: run_at).process