How to reliably process a queue?

Question

Take this conceptually simple task: consuming a queue, sending an email for each entry.

A simple approach would be:

while true:
   entry = queue.pop()
   sendMail();

The problem here is, if the consumer crashes after popping but before/during sending the mail, a mail is lost. So you change it to:

while true:
   entry = queue.peek()
   sendMail();
   queue.pop();

But now, if the consumer crashes after mailing, but before popping, the mail will be sent again when the consumer comes back up.

What's the best-practice way of handling this problem?

Sending email is just an example here which be substituted for any mission critical task. Also assume that the popping of the queue is the only record of the mail having been sent, so the mail subsystem does not record anything itself.

Answer 1

Doesn't your requirement seem like trying to solve the two generals problem (which doesn't have a deterministic solution/limit)? https://en.wikipedia.org/wiki/Two_Generals%27_Problem

Peek - Process - Remove

You want to only remove when you've ensured successful processing, and ensure proper removal. Well any of those messages can be lost/programs can crash at any step.

Most robust messaging queues rely on a set of acks + repeated tries (deliveries) to get the desired behavior (until the acks come back).

But it's actually impossible to guarantee perfect behavior in every scenario. You just have to end up weighing odds and make an engineering compromise between repeated (atleast attempted) processing and "never" (infinite memory etc) losing a message - specific to your actual application needs. Again not a new problem :), and unlikely you'll need to write scratch code for it - like i mentioned, most MQs solve this exact problem.