Is there a way to tell a sleeping delayed job worker to process the queue from Rails code?

Question

I'm using delayed_job library as adapter for Active Jobs in Rails:

config.active_job.queue_adapter = :delayed_job

My delayed_job worker is configured to sleep for 60 seconds before checking the queue for new jobs:

Delayed::Worker.sleep_delay = 60

In my Rails code I add a new job to the queue like this:

MyJob.perform_later(data)

However, the job will not be picked up by the delayed_job worker immediately, even if there are no jobs in the queue, because of the sleep_delay. Is there a way to tell the delayed_job worker to wake up and start processing the job queue if it's sleeping?

There is a MyJob.perform_now method, but it blocks the thread, so it's not what I want because I want to execute a job asynchronously.

Answer 1

Looking at the delayed_job code and it appears that there's no way to directly control or communicate with the workers after they are daemonized.

I think the best you can do would be to start a separate worker with a small sleep_delay that only reads a specific queue, then use that queue for these jobs. A separate command is necessary because you can't start a worker pool where the workers have different sleep delays:

1. Start your main worker: bin/delayed_job start
2. Start your fast worker: bin/delayed_job start --sleep-delay=5 --queue=fast --identifier=999 (the identifier is necessary to differentiate the workers daemons)
3. Update your job to use that queue:

class MyJob < ApplicationJob
  queue_as :fast

  def perform...
end

Notes:

• When you want to stop the workers you'll also have to do it separately: bin/delayed_job stop and bin/delayed_job stop --identifier=999.

• This introduces some potential parallelism and extra load on the server when both workers are working at the same time.

• The main worker will process jobs from the fast queue too, it's just a matter of which worker grabs the job first. If you don't want that you need to setup the main worker to only read from the other queue(s), by default that's only 'default', so: bin/delayed_job start --queue=default.