I would like to efficiently consume events from multiple MQTT topics in a single route in Apache Camel. The Paho MQTT component seems to support only 1 topic at the moment, while I had expected that you could include multiple topics separated by a comma like with the Kafka component.
Kafka Component path parameter
For example, suppose you have the following topics:
but you are interested in all topics except topic a/b/c40.
One option is to use MQTT topic wildcards and drop the events from topic a/b/c40. However, the problem is that c40 takes up a lot of bandwidth (95+ %) and this extra load should be avoided.
Another option is to create a consumer route for each topic and connect these routes to the same destination route. However, creating and managing 40 mqtt clients does not seem ideal to me or is the overhead negligible?
A third option is to initialize the MQTT client with client.subscribe(topiclist) before starting the route and then pass the MqttClient via the advanced option 'client'. Paho Component client option But I am unsure when the client.connect(connectOptions) method should be executed. If this connect method is not executed before starting the route, I receive a 'client not connected' error. But when I execute it before starting the route, then it seemed that the route was no longer receiving all messages.
Which option do you think is best? In the case of the 3rd option, when should the client.connect method be called? Before/After starting the route, in the route ... ?
In my case I went with the second option. It is unfortunate but yes the Paho MQTT client on Camel does not support multiple topics in the same route. I have had this exact issue few months back where I had data coming in over MQTT from multiple (10+) production lines and around 5-6 topics per line with decent sized messages (don't remember the exact size but around 2 - 3 KB per message).
I created a configuration CSV and a small gawk script to generate the camel blueprint, this made the management of the routes very simple. I used one camel context per production line. This data was later processed in one destination topic. Performance was not an issue and the application was able to handle messages coming in every 10 - 15 ms.
I usually keep two settings in mind when dealing with high frequency large payloads:
Only draw back when having so many routes and contexts is that they take some time to start up, but that is not something you would have to do often so its acceptable.