I'm building a database + tool that scours news feeds for a certain term. For example "food poisoning from nuts". I want to scour social media sites, news sites, major news aggregators, etc... for that term.
Edit Added schematic: alt text http://koopics.com/news_parser.jpg
Do you know Yahoo! Pipes? It's a very flexible feed aggregator, and you can manipulate it using YQL, which is quite powerful and has a Python librabry, python-yql :). YQL also has specific "tables" for Twitter and other services and news sources, so, depending on what you want, you might not even need Pipes.
There's a quick example of Pipes + YQL usage at http://blog.ouseful.info/2009/04/27/using-yql-with-yahoo-pipes/ , but you can play around with them at the YQL Console.
So, that's your "Question 1". But, using YQL, "Question 2" (if I understood it correctly) comes naturally, as you use it almost like regular SQL, imposing limits, ordering etc.
