Search code examples
spring-integrationspring-cloud-function

Multiple Spring Integration flows as workers to consume from many SQS queues


I have many SQS queues to which I have to attach workers to consume and process messages (using spring integration flow). My initial thought is to have a single spring boot application with 1 channel per SQS queue(spring integration aws).

This will be like 1 SQS -> 1 Channel -> 1 Flow

However, this might get into maintainability, deployment, memory issues and we won't be able to scale each worker independently. It sounds like a monolith of workers. Another option is to create a spring cloud function per worker type and deploy them as lambda.

Is there any other solution for this scenario in the spring stack (I want to use spring integration and reactor), such that we can scale every worker independently and also manage deployment of every worker independently (may sound like a microservice for each worker, but it does not have a domain, just some processing logic like validations, invoke some api, store to database)


Solution

  • Even if it doesn't have any domain and REST endpoints, you still can treat it as a microservice since it is activated when the message is consumed from the SQS queue. Therefore your direction with Spring Boot + Spring Integration AWS is correct. You just expose an SQS queue as a configuration property and deploy your application into a desired environment with appropriate scaling policy.

    Yes, you probably still can write a Spring Cloud Function for AWS Lambda, but no one stops you to do a Spring Integration logic in the function body.