I need to store files during 30 minutes (or less), then I need to delete it (after processing or by timer).
Non-functional requirements
It must be free and it is desirable to have .net adapter.
loading:
First milestone: 10-20 new files per minute.
Second milestone: 100-200 new files per minute.
Third milestone: 1000-3000 new files per minute.
File size: <10mb - 90%, >10mb - 10%.
I need two solutions: hard storage (file system or db) and memory storage.
What do you advice me?
PS: currently, I research MongoDB.
Interesting problem.
If you do some maths first and look at the max you will need to support:
3000 files per min is 50 per sec on average, if we assume an average file size of 5 MB, that is 250 MB. Then all files will not come at the average speed, so you will need to support atleast 1 GB per second.
At that speed just pushing it across the network becomes a problem.
You therefore need to spread your data across many database servers and network segments.
This spliting of databases is called sharding, MongoDB supports sharding. For others see: http://en.wikipedia.org/wiki/Shard_(database_architecture)