I have a task that renders large sophisticated PDF reports that includes complex charts, SVG images, tables and other data. Processing of 1 request takes up to 10 mins to execute. Sometimes we have to handle hundreds of such requests in limited time-range. Net app that performs this work is currently hosted on single machine. Service is completely stateless. Parallelism on 1 machine has almost no effect so we're looking for ways to scale it horizontally.
I use AWS to host our services. Currently I think about including EC2 instance with this service to manual or auto scaling group with possibility to run required number of instances. Also I think about adding a queue that will carry all requests. So that all instances would work with 1 queue and each instance would process own request (processing of request = rendering 1 PDF report).
Could you please recommend some out-of-box ways / best practices of scaling out CPU intensive work for stateless .NET applications using AWS.
Thank you in advance!
You have the right idea. The requests to process the reports should be put into an SQS queue. Worker EC2 instances can consume those requests from the queue and generate the reports. Put the worker EC2 instances into an autoscaling group that is configured to scale up/down based on the size of the SQS queue.
At a high-level to get started, modify your code to publish/consume requests from the SQS queue, create an instance with this code, and then create an AMI from this instance. Use this AMI in the autoscaling group.
A best practice is to use IAM roles for the instances to manage AWS keys for accessing the queue (and also if you want to persist the reports to S3 or do something else with them).