How a publisher can share data among multiple subscribers?

Question

Every data item published cannot be dropped. So, every data item is important.

3 Publishers and 60 subscribers, are distributed.

First publisher provides data item to 20 subscribers with data item of type:

type struct{
   name  string
   age integer
   gender boolean
}

Second publisher provides data item to another 20 subscribers with data item of type:

type struct{
   ledStatus  boolean
   socketStatus boolean
}

Third publisher provides data item to another 20 subscribers with data item of type:

type struct{
   range integer
   boundary integer
}

---

What should be the communication mechanism , for any publisher to send data to multiple subscribers? subscriber prefers getting notified on new item, to avoid polling traffic....

Answer 1

What you are looking for is a distributed transaction log. One that settles into a single serialised stream, with each message indexed by a large incremented identifier say a 128bit integer.

When a consumer goes offline, it can register itself as part of the registration process request all messages since the last message it processed by providing the id of that last processed message.

The producers emit their message to the log. They are responsible for checking and ensuring that their message has been accepted into the log and reattempting if it was rejected (a rare occurrence but possible).

The distributed transaction log itself could be provided by a central database (databases are essentially accelerated transaction logs), or you could implement your own following an algorithm/protocol such as RAFT..