Search code examples
amazon-web-servicesperformanceherokuamazon-ecsaws-fargate

ECS Fargate with extreme poor performance when querying to Postgres RDS


I'm migrating my Django + Gunicorn + Nginx web apps from Heroku to AWS and the performance now is much slower.

In Heroku I have a microservices structure with many different apps, all of them accessible over internet (public). All the apps are connected to a DB instance hosted in AWS RDS (the communication between the DB and the apps are also made over internet.

To make my structure more secure, I started migrating everything to AWS using ECS Fargate in private subnets and also moved the RDS instance to a private subnet. The microservices are divided in different clusters and services and the communication between them are set with Service Discovery. I configured AWS API Gateway + internal Network Load Balancer (NLB) to be able to access some endpoints from my own computer. Until there everything is working fine, I'm able to access one service from another and also made some tests triggering endpoints from my own application and tested the network route with ubuntu's traceroute command. Everything works quick and sharp.

Then, I started to test a few more endpoints that gets a lot of data from my Postgres database in RDS. I was expecting they would be much more faster compared to the structure in Heroku, since now I have everything in the same network (VPC and subnets). But the reallity s that it takes 5x more time to complete a GET request in the same endpoint with the same hardware resources (0.5 vCPU and 1GB ram).

  • Heroku's request takes 4 secons to complete
  • Fargate's request for the same endpoint takes 20 seconds to complete

Looking into the documentation I found that when upgrading the resources in Fargate, the network performance would also increase. Keeping that in mind I upgrade my task definition to 4vCPU and 8GB ram.

With that I really got my network's performance increased, it now takes 10 seconds (against 20 with the previous resources) to complete the request but it is still really slow compared to Heroku.

CONCLUSION:

  • Heroku takes 4 seconds per request, communication via internet and a 'weak' machine (0,25 vCPU and 512mb RAM)

  • AWS Fargate takes 20 seconds per request, internal communication in the same network and using more resources than Heroku (0,5 vCPU and 1gb ram). When upgrading to 4vCPU and 8gb ram, it still takes 10 seconds (2,5 more than Heroku)

Is there something I could be missing here? Is Fargate's performance bad compared to Heroku? What could be the cause of this slowness?


Solution

  • The app had a misconfiguration with Nginx. I removed Nginx and everything started working as expected.