Search code examples
mysqlamazon-web-servicesinnodb

AWS MySQL 5.6.40 - once overwhelmed, always overwhelmed

My MySQL DB is overwhelmed and I have no idea how to fix it :/

I can have normal load on my DB as long as I want, but as soon as the DB is overwhelmed once, it does not seem to get normal again until a DB restart. Even if I reduce load under the normal load. eg.:

  • 100% load - DB works fine.
  • 400% load - DB gets overwhelmed
  • Back to 100% - DB is still overwhelmed
  • Even 50% load - DB is still overwhelmed
  • Restart DB -> 100% load - DB works fine.

Seems like something is triggered during the 400% load time which causes serious issues in MySQL. AWS metrics/logs do not look suspicious. 'show engine innodb status;' does not look suspicious neither. AWS CPU load: ~2%.

I now recognized that free memory is the reason for this issue. After free memory is empty and swap kicks in, MySQL starts slowing down. Any idea how to avoid this?

Memory Metrics

Solution

  • Seems that the issue was the AWS host system. I made a backup of the DB, and imported it in a new (AWS) DB. Now everything works great again.