handle concurrent access in multiple job queues with multiple workers

Question

I've to design a job scheduler for multi-tenant app. Each tenant will have it's own job queue for processing background task. There are N workers each of which listen to all the queues and take up the job when idle.
eg.
queue 1 : task - A, B, c
queue 2 : task - D
queue 3 : task - E, F
and I have 3 workers w1, w2, w3, all of which listen to all the queues. This whole design is going to be implemented in aws.

It is important that one job is processed only once. Since all the workers are reading queue's, how can I prevent simultaneous access of 1 job to many workers ?
Also if the workers read all queue sequentially then it will keep dequeuing only from first queue till empty, how to handle this situation ?

I initially thought of using sns ntoification when new task is added to job queue, but since all workers will receive it, the core problem won't be solved.

Answer 1

For the first concern, SQS handles distributing tasks to individual workers automatically, go read about Visibility Timeouts.

If you want to maintain separate queues, you need to put the logic in the workers to do the queue switching, basically putting in an infinite loop that is looping over the 3 queues, checking for new work, and only processing a single chunk / message before switching to the next queue:

while (true)
    for (queue : queues) {
        message = getMessage(queue)
        if (message != null) 
            processmessage(message)
    }
}

Make sure you aren't using long polling, as it will just sit on the first queue.