I updated an application that subscribes on MQTT topics and persists received messages in a MongoDB database.
Is use the paho Java MQTT driver and Spring Data MongoDB.
The application works like this:
When a message arrives the message processing is moved to a separate Thread (using ThreadPoolExecutor). Thus arrived messages are confirmed very fast, which I need to not get into trouble with dropped messages because of 'inflight queue full' errors when the system is under temporary heavy load. What might happen, but which I normally not observe, is, that the ThreadPoolExecutor Queue grows rapidly because it can not be handled fast enough.
After updating the system I observe that - especially after the start, but not only - the ThreadPoolExecutor queue grows. The application always manages to catch up with the processing, but there are always times when the queue grows to several 10 thousand tasks. And this with a continuously same load of about 700 messages per second (because of testing).
What I tested so far is:
Java 8 and App build with Java 8 -> works great
Java 11 and App build with Java 8 -> works great (some Illegal reflective access warnings of course)
Java 11 and App build with Java 11 -> makes some problems
The application update contains dependency updates of course, mainly the following
spring-boot-starter-data-mongodb 1.15.12.RELEASE -> 2.2.4.RELEASE
spring-data-mongodb 1.10.11.RELEASE -> 2.2.4.RELEASE
mongodb-driver 3.4.3 -> 3.11.2
org.eclipse.paho.client.mqttv3 1.2.0 -> 1.2.2
MongoDB Server Version is 3.4.24
I suspect that something in the libraries is causing the performance to degrade. Something like the changed WriteConcern in the MongoDB Java Driver like described here (but this is not my problem because I already use the mongo driver 3.4.3 in my Java 8 build application). I just can't find what it can be.
If you have any idea what could be causing this behavior, please let me know. Right now I'm at a loss.
It turned out that the update of the paho java MQTT client from version 1.2.0 to 1.2.1 (or also 1.2.2) caused a significant increase of CPU usage.
The issue #754 was created in the GitHub issue tracker for this.