I have been playing around with Twitter get_followers API requests for a while. I would like to scrape the followers of a particular Twitter id that has 100000 followers.
I have added wait_on_rate_limit = True argument at the API instance creation in order to automatically handle all the wait rate limit issues.
api = tweepy.API(auth, wait_on_rate_limit= True)
I have configured the count variable as 5000 which is the maximum amount of followers that can be scraped per request as per the documentation.
for fid in Cursor(api.get_followers, screen_name=screen_name, count=5000).items():
ids.append(fid)
When I go through the documentation, it has been stated that a maximum of 15 API requests will be allowed in 15 minutes.
So according to my calculation. 75000 followers need to be scraped in 15 minutes. In order to scrape out 100000 followers, it should possibly take less than 30 minutes. But, this is not being the case while I am running the code. It takes multiple wait times. I don't know why is this behaving in this way. Can anyone help me out?
Cursor(api.get_followers, screen_name=screen_name, count=5000).items()
For this get_followers API, the maximum count per page is 200. So, if you set the count variable above 200, it automatically sets to the default value which is 20. That means 20 items per page. This means, only 300 followers can be scraped in 15 mins.
So, set the count variable as 200 which is the maximum allowed count.
Cursor(api.get_followers, screen_name=screen_name, count=200).pages()