Search code examples
amazon-web-servicesaws-lambdacloudserverlessaws-serverless

Automatically suspend whole AWS account after reaching limits


After paying many thousands to AWS due to small programmer's mistake I have a question.

How can I set an action which will suspend the whole account activity after budget alarm happened? For me it seems insane that customer isn't allowed to do such thing easily. We had alarm set up but it was weekend and nobody reacted on it.

We were compensated only 5k(it's a small part of the whole bill).


Solution

  • Suspending the whole account activity isn't going to solve your problem probably. If you started an expensive EC2, and the budget alarm goes off, suspending activity wouldn't mean stopping the instance. It would just prevent any changes from happening.

    Decide on what your goal is first on the cost management of your account.

    For starters, if you haven't already use AWS organizations and set in place some service control policies for expensive services that you don't want to actually use (e.g. redshift). You can also set in place policies to limit the use of large instance sizes as described here.

    Then set in place monitoring using more fine-grained budget alerts. The action on that alert depends on the nature of the cost. An easy start would be to set a daily limit and a montly limit and simply monitor your inbox/slack channel/other ways you receive alerts. Optionally start using cost allocation tags to gain visibilty in what workloads are causing the cost and set more aggressive alerts on untagged workloads so you notice them earlier. In short, I'd suggest first putting the effort into gaining visibility (monitoring) before you automate stuff.

    If you really want to automate stuff, its tricky to give you advice on that because it depends on the nature of the account and workload. If its just EC2 or RDS in a dev environment, you can auto-shutdown using some open source tools or your own scripts. For other services, it depends.