MQ queues vs Batch process : how to replay asynchronous process when it fails

Question

As part of the development of a middleware, I need to choose between two mechanisms to do asynchronous processing (with the possibility of replaying failure cases) I have as ideas the two following techniques:

• Use the MQ queues (during an error, insert a message in the rejection queue and replay it later)

• Use a database to store the cases of failure, and turn a batch to replay them later.

The technical environment is Java (jdlk 8 + wildfly + REST web services as a middleware entry point)

Can you orient me by comparing advantages and inconvinients of the two techniques, and if there is a better solution it will be welcome.

Thanks in advance for your help.

Answer 1

In this situation my preference would be option 1. My reasons would be:

1. No need for an additional DB and the maintenance that it entails (upadating schema, writing code to write and read the messages).
2. You mention using the MQ files, therefore I assume you are already using queues for the messages, if you are using a service bus such as RabbitMQ it can automatically route failures to a failed queue for you.
3. You can spin up multiple competing consumers for the Queue later which will be quicker when clearing down large queues (i.e. better scale-ability). Where as batch process is likely to be a single sequential process that churns through messages one by one (of course this depends on your implementation but is typical).

Additionally if the REST service is temporarily busy you could always implement a retry mechanism on the initial request to give it a better chance of success and then place on the queue if the retry attempts fail for processing later. Try looking at something like Spring retry