I'm looking for the best solution for caching thousands of web pages. Right now I am using flat files, which works great until there's many thousands of flat files, then the entire file system slows down (a lot) when accessing the cache of files (Running on CentOS with EXT3 under OpenVZ). I'd like to explore other options such as Redis or MongoDB as a substitute, but would they be any faster? And if not, what would be the best suggestion?
My system dynamically creates over 40K pages per website, so it's not feasible to do a memory cache either.
Thanks!!
A file cache is fine, you just have to be smart about it. I'd aim to keep directorys to, say, 500 entries or less. with 40k entries, just hashing the url and using the first 2 bytes of the hash will give you 255 folders, each of which should contain on average ~150 files.