Search code examples
amazon-web-servicesmessage-queueamazon-sqsarchitectural-patterns

Amazon SQS: how to find end of job?


Let's say I have a lot of jobs and every job consists from a big number of some elementary operations that many worker applications work upon.

I send a message for every operation to workers, so they can catch these messages, do what is needed, then notify with a message in a control queue some control application.

But how can I know that all operations are done? SQS messages are out-of-order, so I cannot just send a "last" message.


Solution

  • Given not only that messages may be out of order, but that they may fail altogether for various reasons, you generally therefore add both message and job-specific serial numbers if your problem warrants.

    For example; the first job you submit consists of three items, so it is jobs 1-3-1, 1-3-2 and 1-3-3. In this hypothetical numbering scheme Job 1, consisting of three parts, and the three parts. When the master receives 131 and 133 it "knows" that it is waiting on 132 in order to complete the job. If it does not receive it in some duration, it could even reissue the request.

    Some scheme like this would keep your requests and request fragments from getting mixed up.

    Good Luck