I am a newbie who just started reading about distributed system. I am wondering what are some use cases for a distributed queue as opposed to queues on each machine. For example, how RabbitMQ is used among different web servers. How is it used for example in load balancing?
We typically use distributed queues when the up front cost of processing some task is too expensive or otherwise unnecessary. For example, when you upload a video to YouTube, typically there's some processing of the video that occurs before it's displayed on the site. In the modern web, it can be unacceptable for users to have to wait while that processing occurs. So, the video can be stored and a task put on a queue so that processing can take place later. Then, other machines that are polling the queue can process the video at their leisure. This means the user doesn't have to wait for their video to be processed before they can continue on doing other things on the site. It also critically allows for a buffer for periods of high throughput. If users are uploading videos faster than they can be processed by YouTube's servers, the queue grows independently of the back end's ability to process items.
Another consideration is that the distributed nature of the queue allows for fault tolerance. In the YouTube example, that allows the website to respond to the user, assuring the user that their video will eventually be processed. Typically distributed queues have configurable replication levels, where once an item is put on the queue it's guaranteed to be replicated on n nodes and therefore is unlikely to be lost.