Search code examples
pythontwittertweepyrate-limiting

Getting tweets for different hashtags in one call using Tweepy

I want to get popular and recent tweets for different Hash tags, with Tweepy. The solution I have right now is shown below. But this method does a separate call for each hashtag, which is not good in the presence of rate limitations. Is it possible to do this in one call? If so, how?

for ht in hash_tags:
    tweets = tweepy.Cursor(api.search, ht + " -filter:retweets", 
        result_type='mixed', since=date_since, 
        tweet_mode='extended').items(num_of_tweets)
    add_tweets(tweets)
Solution

  • Rather than running a for loop in which you search for one hashtag at a time, you can run a search which contains many of the hashtags you need using the OR operator from Twitter's standard search operators. This is a broader search, and you'll get a mixed bag of results for each hashtag, but it cuts down on the number of requests you make overall.

    So add all of your hashtags to a string, and pass this through as your query.

    #just an example, not the nicest!
    query_string = ""
    for i in hash_tags:
        if i == hash_tags[-1]:
            query_string+=str(i)
        else:
            query_string+=str(i) + " OR "
    tweets = tweepy.Cursor(api.search, q=query_string + " -filter:retweets", 
    result_type='mixed', since=date_since, 
    tweet_mode='extended').items(num_of_tweets)
add_tweets(tweets)

    This means your search will be something like #coolstuff OR #reallycoolstuff OR #evencoolerstuff. It may be worth increasing the number of items returned when doing this, as the search will be so much more broad.

    Keep in mind that there are limits to the size of query you can search with, so you may need to break this down into smaller queries if you've got a lot of hashtags. This may also help you get better results (e.g a more popular hashtag in your query string taking up a lot of your results, so do a separate search for less popular hashtags separately; how you'd measure that though, would be up to you!)

    Hope this helps you get started.