Search code examples
mysqlazuredatabase-designstoragesharding

Mysql database size growing over 4 terabytes ? Azure supports up to 4


A couple of days ago, I was asked for help in this particular situation. A MYSQL Database set up on Azure is reaching 4 terabytes in size. I have set up databases before and developed for them but I'm not really a dba.

The problem according to them is that Azure size limit is 4 terabytes (and it will double that size in a couple of months but luckily just it wont keep growing like that). I talked to them about achieving some of the data, but they need all 10 years worth of data apparently. They don't want to move from Azure or use something other than MYSQL. One thing they pointed out to me was that 1 table in particular was almost 2 terabytes in size.

Unfortunately, I haven't been given access to the database yet but I just wanted to ask about my options in a situation like this. I looked into this a bit and I saw Stuff like MYSQL sharding. Is this the only option ? Can it be done on Azure (I saw SQL sharding articles for SQL server on Azure but not for Mysql). Can I partition some tables into another MYSQL database for example ?

I guess I'm just looking for advice on how to move forward with this. Any link on something like this is appreciated.

Thank you


Solution

  • Simple Answer

    4TB is not the MySQL limit, so Azure is limiting you. Switch to another service.

    Future problems

    But... 4TB is rife with issues, especially for a non-dba.

    • Over-sized datatypes (wasting disk space)
    • Lack of normalization (wasting disk space)
    • Need for summary tables (if it is a Data Warehouse)
    • Sub-optimal indexes (performance)
    • Ingestion speed (bog down on loading of fresh data)
    • Query speed
    • Partitioning to aid in purging (if you will eventually purge 'old' data)
    • Sharding (This is as big a discussion as all the others combined)

    All of these can be tackled, but we need to see the schema, the queries, the dataflow, the overall architecture, etc.