We are currently building a service with pulsar where multiple producers can send messages onto a topic. Now we want the producers to get feedback on the processing of their messages. We thought to have a second out-topic where the producers can subscribe to and our service writes results into. Is it possible that the producers only get messages that are answers to their messages produced on the input-topic? Or do we need to have a separate output-topic for every producer? Or is Pulsar simply not suitable for our use-case?
We searched for a method to let consumers only consume specific messages of a topic, but found no way to do it.
I can think of two different approaches off the top of my head, and which one you use depends upon the sensitivity of the data sent back to the producers to provide feedback on the processing of their messages.
If that data isn't sensitive, you can use a single topic to store these messages. Each producer would have their own Exclusive subscription to this topic to ensure that they receive all of the messages but would only react/process messages intended for them (as indicated by a message property of key, etc.). All other messages would be ignored.
If the data sent back to the producer is sensitive, you can still use a single topic for these messages and use message-level encryption and different public/private keypairs for each producer. This will ensure that only the intended recipient of the message can actually decrypt and read the message contents.
Last but not least, you can also use the topic-per-producer pattern you mentioned to ensure that only the intended receives the feedback messages related to them. While this approach does have the benefit of sending less data from the brokers to the clients vs. sending all messages to every producer like the previous two approaches. It incurs additional overhead for topic management, such as setting up security policies to enforce producer-level access, etc.