Search code examples
cassandraleveldb

why there is limit on column value (2 GB) in cassandra but no such limit in leveldb


There seems to be limit on column value in Cassandra (2GB), however leveldb (a similar design with SSTable and memtable) does not have such limit?


Solution

  • Pragmatically the column value limit is far lower than 2gb in C*, simply from the JVM not able to allocate objects that big on the heap efficiently. In fact post 2.1 you would have failures if a mutation is half the size of your commitlog segments (~32mb default) although you can increase that to make it work, everything would perform horrible. That said if you have smaller values split up into rows, you could end up having partitions into the GB or even 100s of gb but you would need to be willing to take a significant performance hit and tune for it. Not to mention the hotspotting difficulty.

    For that matter leveldb does poorly with large values as well (ie 100k is considered large and it will start crawling with a few hundred ops/sec). It is usually recommended to keep values sub 10mb or 1mb.

    Most likely neither will be able to work (at least at any usable rate) with anything near 2gb. Thats not the kinda thing databases store.