Search code examples
rtwitternullrate-limiting

Twitter rate limit changes to NULL, R tweetscores package self-terminates

I am using the R tweetscores package to estimate Twitter users’ ideology score (i.e. estimating a user’s ideology based on the accounts they follow). I am using the code below to loop through a list of usernames, get who they follow (getFriends()) and then estimate their ideology score (estimateIdeology2()). The getFriends() function makes calls to the Twitter API until it hits the rate limit. In this case, it should wait and then resume to making calls. However, the loop seems to self-terminate after about 40 minutes. It looks like the variable that holds the number of calls left, changes from 0 to NULL after a while, causing the loop to break. Has anyone encountered this and/or knows how to fix this issue? I have tried adapting code to catch it when this variable turns to NULL and change its value but that doesn't prevent the loop from terminating. I would ideally like to keep this loop running and not manually restart it every 40 minutes. The raw code for the getFriends() function is here (it seems to break at line47): https://github.com/pablobarbera/twitter_ideology/blob/master/pkg/tweetscores/R/get-friends.R

for(user in usernames$user_screen_name){
  skip_to_next <- FALSE
  tryCatch({
    friends <- getFriends(screen_name=user, oauth=my_oauth)
    results <- estimateIdeology2(user, friends)
  }, error=function(e){skip_to_next <<- TRUE})
  if(skip_to_next) { next }     
  print("results computed successfully.")
  user_scores[nrow(user_scores) + 1,] = list(screen_name = user, 
                                             ideology_score = results)
}
Solution

  • Tweetscores package uses API v1 endpoints and rtweet package. These are being replaced by API v2 and academictwitter. So I would suggest you to get friends list through academictwitteR.

    get_user_following(user_ids, bearer_token)

    But rate limits are real: You can make 15 requests per a 15 minute window. So if your users only follow a handful accounts (so that no pagination is required), in the best case scenario, you can get followers for one user per minute. If you have got hundreds of thousands of users, this could take ages. Looking ways to work around this issue.