Mechanism for reading all messages from a queue in one request

Question

I need a solution for the following scenario which is similar to a queue:

I want to write messages to a queue continuously. My message is very big, containing a lot of data so I do want to make as few requests as possible. So my queue will contain a lot of messages at some point. My Consumer will read from the queue every 1 hour. (not whenever a new message is written) and it will read all the messages from the queue.

The problem is that I need a way to read ALL the messages from the queue using only one call (I also want the consumer to make as few requests to the queue as possible).

A close solution would be ActiveMQ but the problem is that you can only read one message at a time and I need to read them all in one request.

So my question is.. Would there be other ways of doing this more efficiently? The actual thing that I need is to persist in some way messages created continuously by some application and then consume them (also delete them) by the same application all at once, every 1 hour.

The reason I thought a queue would be fit is because as the messages are consumed they are also deleted but I need to consume them all at once.

Answer 1

I think there's some important things to keep in mind as you're searching for a solution:

1. In what way do you need to be "more efficient" (e.g. time, monetary cost, computing resources, etc.)?
2. It's incredibly hard to prove that there are, in fact, no other "more efficient" ways to solve a particular problem, as that would require one to test all possible solutions. What you really need to know is, given your specific use-case, what solution is good enough. This, of course, requires knowing specifically what kind of performance numbers you need and the constraints on acquiring those numbers (e.g. time, monetary cost, computing resources, etc.).
3. Modern message broker clients (e.g. those shipped with either ActiveMQ 5.x or ActiveMQ Artemis) don't make a network round-trip for every message they consume as that would be extremely inefficient. Rather, they fetch blocks of messages in configurable sizes (e.g. prefetchSize for ActiveMQ 5.x, and consumerWindowSize for ActiveMQ Artemis). Those messages are stored locally in a buffer of sorts and fed to the client application when the relevant API calls are made to receive a message.
4. Making "as few requests as possible" is rarely a way to increase performance. Modern message brokers scale well with concurrent consumers. Consuming all the messages with a single consumer drastically limits the message throughput as compared to spinning up multiple threads which each have their own consumer. Rather than limiting the number of consumer requests you should almost certainly be maximizing them until you reach a point of diminishing returns.