During an investigation into Spring Cloud Data Flow, I observed that the SCSt httpclient processor app will process just one message at a time. This leads to low throughput and low CPU utilisation if the response from the endpoint is delayed. For example, a latency of 5 seconds results in a maximum throughput of 12 messages per minute (assuming a single consumer.)
Is it possible to patch or configure the httpclient to act in a non-blocking fashion? Rather than idling, waiting for a response, can the httpclient process the next message(s) in a loop until it receives a callback for the response?
An async client would risk message loss since the incoming message will be acknowledged before processing is complete.
Increasing the ....consumer.concurrency will add more threads and increase throughput.
If you are using the Kafka binder, you need at least as many partitions as the concurrency setting.