I'm investigating a tech for our cluster. Pulsar looks good, but the usage looks more like a queueing system. Of course, queueing system is good to have, but I have a specific requirement: broadcasting.
We would like to use one machine to generate the data and publish it to a Pulsar topic. Then we use a group of servers, forming a replica. Each server consumes the message flow on that topic, and serves clients via WebSocket.
This is different than the Shared subscription, because each server needs to receive all messages, not a fraction of it.
I came to this post: https://kafkaesque.io/subscriptions-multiple-groups-of-consumers-on-pulsar-topic/ , which explains how to do such a job: each server needs create a new exclusive subscription, say use a UUID as its subscription name, from the unique exclusive subscription you can get the full message flow of that topic.
But since our server replica can be dynamic, so once some of the server restart, they will create new UUID subscription again, which will leave many orphan subscriptions on the topic, which eventually would become maintenance headache.
Anyone has the experience to setup a broadcast use case using Pulsar?
Using an exclusive subscription for each consumer is the only way to ensure that each of your consumers receives ALL of the messages on the topic, and Pulsar handles multiple subscriptions quite well.
The issue it seems is the server restart use case, and I don't think that simply connecting with a new UUID subscription is the right approach (putting aside the orphaned subscriptions). You really want to have the server reuse the previous subscription after it restarts. This is because each subscription keeps track of the last message in the topic that it had processed and acknowledged, so you can pick up exactly where you had left off before the server crashed if you reconnect with the same subscription UUID. If you connect with a new UUID, then you will start processing messages produced from that point in time forward, and all messages produced during the restart period will be "lost"
Therefore, you will need to find a mechanism to share these UUIDs across server failures and return them to the restarting server. One approach would be to have a mechanism similar to zookeeper leader election, in which each server is granted an exclusive lease that expires periodically. The server must then periodically refresh the lease to retain it. Then if the server were to crash, it would fail to refresh the lease on that UUID and the restarting server would then be granted the lease when it attempts to reconnect.
See https://curator.apache.org/curator-recipes/leader-election.html for a better explanation of the pattern.