I have a Rails 3 application. One of the controller method is for parsing large number of Twitter search results and storing them to the database. All works fine if the number of URLs to parse is small. But once the URLs reach 1000 or more, I have the following error after a few seconds (copied from the log file):
Address: http://search.twitter.com/search?q=+Chas%20Salon+near:%22Baltimore%22+within:15mi Completed in 111436ms
OpenURI::HTTPError (420 unused):
app/controllers/twitter_reviews_controller.rb:41:in block in new'
app/controllers/twitter_reviews_controller.rb:20:ineach'
app/controllers/twitter_reviews_controller.rb:20:in `new'
I am using Hpricot to parse the Twitter search results.
The line number 41 in the above message is this:
doc = Hpricot(open(address))
Does anyone knows where is the problem?
Thank you very much for any help.
Cheers,
Tony.
You are being rate limited by Twitter server. read more here: http://dev.twitter.com/pages/rate-limiting