MQ-like Java library/framework for processing large data with fault-tolerance and re-injecting of large messages

Question

The requirements are: several services produce lots of requests with data streams (1-10 megabytes per request) and several consumers process these requests atomically.

So the request data can not be split. The key requirement is that if some consumer fails to process request - this request must be re-injected into queue and sent to different consumer. Processing of single request may take tens of minutes. Consumer may 'die' during the processing, so some sort of task monitoring is required.

I considered so far

• http://kafka.apache.org/
• http://storm-project.net/
• http://activemq.apache.org/
• http://activemq.apache.org/apollo/

but neither of them seems to meet my requirements. Any suggestions?

Answer 1

Regarding your requirements (> 1 MB per request, processing > 10 minutes) ... is it still messaging?

I would just put it in a database (RDBMS). If polling is not acceptable, I'd consider triggers/stored procedures in the database. Stored procedures written in Scala work perfectly in Oracle and PostgreSql.