I don't know how to put this, but I am investigating a way to delay the writes to my MySQL database. My application does data mining from twitter and it receives like 2800 tweets per minute and most of them are new and has to go to the database as new inserts while the left tweets are not new and goes to my database as updates. Meaning that my application is writing and reading from the database all of the time causing the disk to be at 99% of usage and making the application a bit slow and bottle-necked. I am using EclipseLink with weak soft caching and it works fine for the reads but I need something for the writes too to make them stay in memory for a while and flushed all together in just one access. My question is if there is some framework or a middleware that does this automatically. I think I can rewrite my code to accumulate my entities until reaching some limit and persist them all together in one transaction but it seems to me that it is something that could be already implemented to be done transparently from my code.
You might consider introducing Coherence or a similar product into your stack. Your requirement is covered with Write-Behind caching.