Message Broker - dependency between jobs

Question

I'm trying find a good queue server / message broker that can give me the ability to push jobs to the queue but also to:

1. Make dependencies between jobs (e.g. Run job B only after job A has finished)
2. Allow rerun task if the subscriber failed to execute it (without push it back to the queue from the subscriber)
3. Persistancy (in case of server restart etc)
4. Scaling (to be able to add more nodes in case the server is loaded)
5. Advantage: managed solution in AWS

I know a lot of names like RabbitMQ, ActiveMQ, Kafka but i want to hear from real-life experience and not just from articles i already read.

Answer 1

I also had the same requirement where I evaluated different Queue services like RabbitMQ and Apache Kafka, but all these services, require us to maintain our own servers and manage it. The problems with this is that it's costly to maintain our own servers, and we need to manage it ourselves on scalability (unless we use a server that provides scalability).

Then I switched to AWS SQS, with AWS Lambda which was ideal for my requirement. The main advantage is that, its completely serverless, and AWS will handle scalability of it. We need to keep track of the limitations in each of the services, which will only get affected if we scale into a very large level.

Now, let's look how this solution will cater to your requirements.

1. Make dependencies between jobs (e.g. Run job B only after job A has finished)

When a message is received by SQS, you can invoke a Lambda function (Job A) and let that invoke (Job B) to ensure the order is maintained. A similar approach can be used without Lambdas, where you use you can direct the messages in the order you want after each job is completed. The advantage of using AWS SQS and Lambda is that, SQS provides the functionality of invoking lambdas (a feature introduced in June 2018), where we don't need to poll the queue every time.

2. Allow rerun task if the subscriber failed to execute it (without push it back to the queue from the subscriber)

If a subscriber fails it, you can send the message to a Dead Letter Queue (DLQ), which is another SQS Queue, that stores messages which were failed by the receiver. (See this link). With this, you can use a similar approach as mentioned in 1. where you can let the messages in DLQ to process separately.

3. Persistence (in case of server restart etc)

SQS persists your messages for a given period of time. You can specify the period you want to store the message in the queue. If you want hard persistence, which does not expire after a certain time, consider storing it in a a database or other storage mechanism like S3.

4. Scaling (to be able to add more nodes in case the server is loaded)

By default AWS provides high scalability, where certain limitations are set for each service. Be fully aware of the limitations, where it will only exceed if we go into a very large scale. (this can always be increased by contacting AWS support team)

5. Advantage: managed solution in AWS

As specified above, these AWS services are managed by AWS itself. So there is nothing to worry about it, as long as you stay within the limits.

All of the solutions you have mentioned in the question is good, but the usefulness of it depends on the use-case scenario. According to the requirements mentioned, AWS SQS will be ideal for this scenario.