I have been asked to look into developing a solution for logging for a large organisation. I was thinking having some sort of service that has a msg queue of logs and inserts them into some sort of database but there is lie's the rub.
Oracle and SQLServer don't seem to be able to handle very large amounts of data and performing quick searches on them. They seem to be more focused on having large number of transactions doing small things than small number of transactions doing big searches. Would you be able to give me a nudge in the correct direction on some software / databases we could implement for this specific purpose?
The answer is not Relational DB's.
Look into http://hadoop.apache.org/