Spring Reactor benefits when Number of Publisher threads is much higher than number of consumers

Question

I have the following use case:

• N threads publish data (N can range from 10 to 1000 threads), those thread can make HTTP request, jdbc calls, pure java processing using only local computer
• 1 to M thread consume it making IOs (send HTTP requests, write to Database... possibliy in bulk) , those threads should not slowdown the publishers. M must not exceed 10 threads.

N threads may publish data much faster than consumer will consume it, but the idea is to minimize the slowdowns of publisher.

I have implemented an approach based on ArrayBlockingQueue where publisher write, and a thread that takes data our of the queue and processes it, it works but results are not great.

I am thus studying the Reactor pattern and particularly Spring-Reactor to see if it could be a response to my use case. It is the case ?

I read :

• https://spring.io/guides/gs/messaging-reactor/#initial => This one does not seems to meet my use-case.

• https://github.com/reactor/reactor/blob/master/reactor-core/src/test/java/reactor/core/processor/ProcessorThroughputTests.java => Seems closer to mine but need confirmation

In my situation where number of publisher threads is much higher than number of consumers is it the right choice ?

Answer 1

It sounds like you might want to look at Reactor's PersistentQueue facility and separate your Publisher from your Subscriber across that. It's a normal Queue implementation but it uses the Chronicle Queue for persistence, fail-over, and replayability. It is also extremely, extremely fast.

You would basically have publisher pushing data into the PersistentQueue from one side and a set of subscribers pulling from it on the other. It might be a drop-in replacement for your current use if you're already using a Queue.

I need to write a wiki page on it to show some basic use patterns.