Search code examples
avroapache-pulsar

Can input and output topics be the same for a Pulsar Function?


Is it an anti-pattern to have the same input and output topic when using Pulsar functions?

In my case, I have been using just one topic where I have a Cassandra sink consuming the messages. I was thinking to create a function that will read messages from this topic and send the transformed messages to the same one. The sink will be able to sink into Cassandra just the processed messages because they will respect the schema.

Is this a bad practice?


Solution

  • I would not recommend it. You'll need to filter the transformed messages in your function otherwise you get an infinite loop. Also the sink will have to filter the raw messages. These filters are wastes of resources. It would be far better to have distinct topics for the raw and the transformed messages. Is there something preventing you to do it ?