We are trying to put several Tb in to MySQL Cluster, unfortunately the index does not fit in to memory.
Are there way to overcome this limitations of mysql?
are there way in mysql process range operations in parallel?
My data has a 3D points: (id x y z idkey someblob) inside the MYisam with 128 partitions. The NDBCLuster was unable load the data due to memory limits.
indexing goes over idkey(this is pre calculated peano-hilbert key).The total row count is about 10^9.
Thanks Arman.
EDIT my setup is 2 datanodes 2 mysqld one mdm. 8Gb RAM per ndb with 4 cores.
The whole system has 30Tb Raid6.
The system is linux Scientific LInux 6.0, the cluster is 7.1 compiled from source.
It sounds like MySQL is ill-suited for the task (sorry). I would check out Tokyo Tyrant, maybe MongoDB or any other distributed key-value storage system. There are also specialized commercial products.
MongoDB is able to swap out some of it's indexes to the HD. I guess your problem is that MySQL just can't do that (I'm not a MySQL-guy though).