I might have gone down the path of over optimizing too soon and gotten myself lost. I'm recording all the moves possible in a board game's game tree. I have little hope of completing the tree since it will get so big (10^28) but want to get a good chunk if possible. In anticipation of slow queries I split the tables into ~50 branches of the tree with suffixes describing each branch.
Unfortunately my application has lots of reads, writes, updates and joins so things got slow quickly before I split them up. Since then I've also added some very helpful indexes which might have solved initial sluggishness. However as the app develops it's getting more and more complicated to switch between so many tables with more complicated joins. I've recently heard about using a master slave set up as well as a merge engine to help with large tables. Did I choose the wrong solution to my problem or should I just stick it out?
10^28 rows or anything else, to put it bluntly, impossible. Calculate the cost of that much disk space; that should scare you. You need to focus on "pruning" your trees.
PARTITIONing
looks tempting, but it does not inherently provide any performance benefit (with rare exceptions). Ditto for manually partitioning into 50 tables. MERGE
is merely an old variant of PARTITION
. Replication might help for read scaling. Sharding (splitting data across multiple machines) could help, but adds to the cost and complexity -- and still won't get you to 10^28 of anything.
If you provide SHOW CREATE TABLE
and some of the queries, we can discuss optimization, bit, indexing, etc, techniques. These may help you some.
Are you using 64-bit BIGINT UNSIGNED
for keeping track of some things on an 8x8 board? And using boolean arithmetic for manipulating them? In some situations, this can significantly decrease the disk space, the number of queries needed, etc.