My company is using CloudFront to serve it's services for over a year. It is supposed to redirect trafic to a Load Balancer, which distributes load to the ECS. Last night, all of a sudden certain IP addresses, like our office ip address, call center ip address, started to receive a CloudFront HTTP 504 Gateway Timout errors. If I switch to mobile internet over my phone - everything seems to be fine. The execution time is not long at all - one of the services is a simple fornt-end only website.
The same happened without tinkering with a settings or anything. In addition, the same happened to our Production and Development environments, which are using different AWS accounts. The WAF is turned off, so it should not be an issue here (dev environment never had it in a first place).
Most importantly some of our integrations stopped working for this very reason, therefore it is critical.
I would appreciate any help.
There is a auto-cloudfront security group, that whitelists CloudFront edge servers ip addresses, so Load Balancer would accept only connections from them. There is a lambda function to automatically update Security Group with new ip addresses.
On the day of issue, AWS added 12 new CloudFront edge servers. In total there was 69 of them. The root cause of issue was that Security Group can only hold up to 60 ip addresses, while there are 69 servers/ip addresses. Therefore the Security Group did not get updated and some the edge servers were not whitelisted.