In order to avoid rate limit error I used the parameter:
wait_on_rate_limit
in function
api = tweepy.API(auth,wait_on_rate_limit=True,wait_on_rate_limit_notify=True)
At first my program worked fine. When I surpass the rate limit I got the message:
"Rate limit reached. Sleeping for: 909". My program was sleeping for this amount of time and then my program continued to collect data. At some point I got some back to back errors.
...
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
...
urllib3.exceptions.ProtocolError: ('Connection aborted.',
ConnectionResetError(10054, 'An existing connection was forcibly closed by
the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
...
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
...
tweepy.error.TweepError: Failed to send request: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
My code:
for user in tweepy.Cursor(api.friends, id="twitter").items():
friendsOfUser=user.screen_name
## Do something with friendsOfUser
Is there anything that I can do?
You can't do anything with the fact that host closes the connection. If you are waiting for the rate limit, I bet you're a bit aggressive in API usage :) Try catching TweepError and explicitly waiting for some time and them try again.
You can try something like this:
import time
...
try:
for user in tweepy.Cursor(api.friends, id="twitter").items():
friendsOfUser=user.screen_name
...
except tweepy.TweepError:
time.sleep(120) # sleep for 2 minutes. You may try different time