Search code examples
databasejsoncouchdbflat-file

How to best store a large JSON document (2+ MB) in database?


What's the best way to store large JSON files in a database? I know about CouchDB, but I'm pretty sure that won't support files of the size I'll be using.

I'm reluctant to just read them off of disk, because of the time required to read and then update them. The file is an array of ~30,000 elements, so I think storing each element separately in a traditional database would kill me when I try to select them all.


Solution

  • If you intend to access specific elements one (or several) at a time, there's no way around breaking the big JSON into traditional DB rows and columns.

    If you'd like to access it in one shot, you can convert it to XML and store that in the DB (maybe even compressed - XMLs are highly compressible). Most DB engines support storing an XML object. You can then read it in one shot, and if needed, translate back to JSON, using forward-read approaches like SAX, or any other efficient XML-reading technology.

    But as @therefromhere commented, you could always save it as one big string (I would again check if compressing it enhances anything).