We have a specific use case where we need to run SQS consumers that compute windowing statistics from incoming messages. The application flow involves running an SQS consumer, parsing the message, and storing it in an in-memory heap. Periodically, a scheduled thread sends a summary of the heap to a downstream sink asynchronously, running every x minutes. However, due to cost and speed considerations, we must maintain the heap in-memory and cannot use an external database.
After exploring the available options, I have narrowed down the choices to three AWS services:
Ideally, we would prefer a serverless solution like Fargate, but we are unsure if it can fully meet our use case due to the maximum runtime constraint as we expect the SQS consumers to run 24/7. We would appreciate insights and advice from the community on which option (EC2 with auto scaling, ECS, or Fargate) would be the most suitable choice for our scenario, considering the in-memory heap maintenance, periodic sink updates, cost, and speed requirements.
This answer is necessarily subjective there are so many different ways to achieve your aims. I'll just weigh in with my own perspective.
You'll first need to decide whether you want to containerize your application. You mentioned that you don't have expertise with containers, so there will be a bit of a learning curve for you to build the container image, set up an image repository (preferably using ECR), configure the container definitions within ECS, etc.
If you decide to go ahead with containerizing, ECS with EC2 or with Fargate will suit your needs. Both can run your application indefinitely (there is no maximum runtime constraint). ECS with EC2 will be the less expensive option in terms of direct cost incurred by AWS, but you're then responsible for maintenance of the EC2 instance. It's up to you to decide whether the total cost of ownership is in the favor of ECS with EC2 or Fargate.
With that said, given your unfamiliarity with containers and preferences for lower cost with limited service interruptions, EC2 with Auto Scaling Groups may be a good choice. You'll have complete control from the operating system on up through the app. It's likely the most cost effective option, esp. with a Compute Savings Plan. You can choose your optimal instance type, possibly a memory optimized instance. Of course, this will be an entirely self-managed solution: you're responsible for security, monitoring, maintenance, etc of the VM.
Any one of the three options you proposed could work. Hopefully this additional color helps you to make a decision.