Search code examples
pythontwittertweepy

Tweepy only returning 1 tweet


Ive got a python script intending to scrape tweets from twitter and append them to a csv file. Im using the tweepy module however it is only returning 1 tweet. Is this a problem with my for loop, or with the call to the twitter API?

for status in tweepy.Cursor(twitterapi.search,q="labour party",since="2018-05-01", until="2018-05-10").items(200):
    if 'RT' not in status.text:
      with open('C:/Users/User/Desktop/twittersentiment.csv', 'wb') as f:
               w = csv.writer(f)  
               favourites = status.user.favourites_count
               location = status.user.location.encode('utf8')
               tweet_text = ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)"," ",status.text.encode('utf8')).split())

               date = status.created_at.strftime('%m/%d/%Y')
               a = [location]
               b=[favourites]
               c=[tweet_text]
               d=[date]
               zip(a,b,c,d)
               w.writerow(zip(a,b,c,d))

Solution

  • You should open the file before you start iterating the tweepy.Cursor otherwise each iteration of the cursor will create a new file with one entry, overwriting the previous file.

    with open('C:/Users/User/Desktop/twittersentiment.csv', 'wb') as f:
        w = csv.writer(f)  
        for status in tweepy.Cursor(twitterapi.search,q="labour party",since="2018-05-01", until="2018-05-10").items(200):
            if 'RT' not in status.text:
                favourites = status.user.favourites_count
                location = status.user.location.encode('utf8')
                tweet_text = ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)"," ",status.text.encode('utf8')).split())
    
                date = status.created_at.strftime('%m/%d/%Y')
                a = [location]
                b=[favourites]
                c=[tweet_text]
                d=[date]
                zip(a,b,c,d)
                w.writerow(zip(a,b,c,d))