I want to check that I am not causing an infinite number of workers to be created on heroku.
I have the following job that is started by the clockwork gem on heroku:
every 5.minutes, SyncAccounts
The Sync Accounts looks like this:
def perform
UserRepo.all.each do |user|
EnqueueFolderSyncJobs.perform_async user.id
SyncContacts.perform_async user.id
SyncMeetings.perform_async user.id
end
end
It iterates through every user and creates further workers. Each of these workers might create more workers, for example, EnqueFolderSyncJobs creates a worker to sync imap folders:
google.email.folders.each do |folder|
SyncFolder.perform_async user_id, folder unless skip?(folder)
end
So every 5 minutes this is happening.
Does calling perform_async mean that the job only happens once?
Do I need call drain at the end of all this?
Also, all the workers are being created under the same queue, should I create separate queues for SyncFolders for example?
Sidekiq by default limits concurrency to 25 workers. You can increase that in your sidekiq.yml if need be. But that concurrency is what limits it from "infinite" workers. https://github.com/mperham/sidekiq/wiki/Advanced-Options#concurrency
Calling "perform_async" means you are calling that job once, it would only occur again if it failed and was retried. I see no reason for you to create another queue, unless some jobs take precedence and should always be processed first. https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
Calling .drain is unnecessary except in test environments.
If you absolutely want your workers to be processed in a certain workflow, instead of having all your jobs dumped into the queue at once (in your example, it doesn't look important), take a look at Sidekiq Superworker. I haven't used it myself but it looks interesting.