DSL Integration Flows with retry mechanism and how it works

Question

I have implemented a retry mechanism which works well based on the following:

https://github.com/spring-projects/spring-integration-samples/issues/237

The application consumes events from kafka, transforms those events and sends them as an HTTP request to a remote service, so it's in the integration flow that sends the HTTP request where the retry mechanism is implemented.

I was worried about sending the requests to the remote service in the same order as they come in from kafka during a temporary failure (network glitch) to avoid an overriding, but fortunately it looks like the order is kept, keep me honest here.

It seems that during the retry process all events coming in are "put on hold" and once the remote service is back up before the last try, all events are sent.

I would like to know two things here:

1. Am I correct with my assumption? Is this how the retry mechanism works by default?
2. I'm worried about the events getting back (or stack) up due to the amount of time it takes to finish the current flow execution. Is there something here I should take into consideration?

I think I might use an ExecutorChannel so that events could get processed in parallel, but by doing that I wouldn't be able to keep the order of the events.

Thanks.

Answer 1

Your assumption is correct. The retry is done withing the same thread and it is blocked for the next event until the send is successful or retry is exhausted. And it is really done in the same Kafka consumer thread, so new records are not pulled from the topic until retry is done.

It is not a correct architecture to shift the logic into a new thread, e.g. using an ExecutorChannel since Kafka is based on an offset commit which cannot be done out of order.