I'm exploring different ways of storing user-uploaded files (all are MS Office documents or alikes) on our high load web site. It's currently designed to store documents as files and have a SQL database store all metadata for those files. I'm concerned about growing out of the storage server and SQL server performance when number of documents reaches hundreds of millions. I was reading a lot of good information about CouchDB including its built-in scalability and performance, but I'm not sure how storing files as attachments in CouchDB would compare to storing files on a file system in terms of performance.
Anybody used CouchDB clusters for storing LARGE amounts of documents and in high load environment?
Why can't you expand your system to mulitple storage servers? If the documents are not stored in SQL, then SQL performance should be able to handle a large table, assuming you index and maintain it properly (of course there are hardware considerations). You'd just need to alter your application so it would start storing in different locations according to whatever rules you define.
SQL 2008 also has a new FILESTREAM function which will increase managebility of the stored files. It looks like SQL will handle all the security and housekeeping on those files.
http://msdn.microsoft.com/en-us/library/cc949109(SQL.100).aspx