Heroku: Why does increasing number of worker dynos increase processing time of job?

Question

I'm seeing strange behavior when scaling Delayed::Job workers on Heroku.

I have a few thousand jobs that are all basically identical. When I assign 1 worker dyno to that queue, each job completes in about 4s.

When I scale the number of workers to 2, processing time averages 8s per job

When I scale the number of workers to 10, average processing time per job increases to above 30s per job.

I would not expect processing time per job to increase when scaling the number of workers.

As it is currently behaving, there is no way to scale up the number of workers to "churn through" a backlog of jobs, as the increase in processing time offsets any gains in having more workers.

Has anyone else seen this behavior and (more importantly) know how to resolve the issue?

Answer 1

Do you have any metrics on the database processing time? Seems possible that the bottleneck could be in the database engine and so no matter how many workers you have, you'd still be locked up there...