I am using MQTT in an IoT application. For that purpose each IoT device will publish data to the IoT broker under their Topic and the processing server would subscribe to some wildcards topic to access the data. This processing server needs to save this data in database. Now the thing is that at one time, thousands or millions of devices will be publishing to the broker. If the processing server accesses the data, before it can finish processing it, it is swarmed by the next data. How can I solve it? I was hoping there could be some queue like mechanism at the client or broker side. If so, how can this be done? Or should I use some other protocol instead of MQTT?
Assuming a constant rate of messages, queueing them up in either the subscribing client or the broker isn't going to help if the subscribing client can not process them quicker than they arrive. That's just going to lead to memory leak that will eventually consume all the available memory and crash the application, probably losing the messages.
MQTT does not normally support queuing messages for online clients. With a client that is subscribed at greater than QOS 0 it can be possible to queue messages in the broker by controlling how the client acknowledges the messages to the broker as part of the QOS handshake, but as I said this will just lead to them backing up in the broker until it runs out of memory. Being able to do this also depends on which client library you are using.
The correct solution is probably to have multiple processing clients and use something called Shared Subscriptions to ensure that each message is only delivered to a single processor. Shared Subscriptions is a feature added to the MQTT spec at v5. More information about Shared Subscriptions can be found here