How do i create a CSV with a simple search on tweepy?

Question

I'm trying to make a script that downloads a twitter search in a .CSV format, but, there a error with my code, any help???

import tweepy
import csv
import pandas as pd
####input your credentials here
consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth,wait_on_rate_limit=True)
#####United Airlines
# Open/Create a file to append data
csvFile = open('test.csv', 'a')
#Use csv Writer
csvWriter = csv.writer(csvFile)

for tweet in tweepy.Cursor(api.search,q="petya",count=100,
                           since="2017-04-03").items():
        print ("ID:", tweet.id)
        print ("User ID:", tweet.user.id)
        print ("Text:", tweet.text)
        print ("Created:", tweet.created_at)
        print ("Geo:", tweet.geo)
        print ("Contributors:", tweet.contributors)
        print ("Coordinates:", tweet.coordinates) 
        print ("Favorited:", tweet.favorited)
        print ("In reply to screen name:", tweet.in_reply_to_screen_name)
        print ("In reply to status ID:", tweet.in_reply_to_status_id)
        print ("In reply to status ID str:", tweet.in_reply_to_status_id_str)
        print ("In reply to user ID:", tweet.in_reply_to_user_id)
        print ("In reply to user ID str:", tweet.in_reply_to_user_id_str)
        print ("Place:", tweet.place)
        print ("Retweeted:", tweet.retweeted)
        print ("Retweet count:", tweet.retweet_count)
        print ("Source:", tweet.source)
        print ("Truncated:", tweet.truncated)

    # Write a row to the CSV file. I use encode UTF-8
    csvWriter.writerow([tweet.created_at, tweet.user.id, tweet.id, tweet.geo, tweet.text, tweet.contributors, tweet.favorited, tweet.source, tweet.retweeted, tweet.in_reply_to_screen_name, eet.in_reply_to_status_id_str('utf-8')])
    print tweet.created_at, tweet.user.id, tweet.id, tweet.geo, tweet.text, tweet.contributors, tweet.favorited, tweet.source, tweet.retweeted, tweet.in_reply_to_screen_name, eet.in_reply_to_status_id_str
csvFile.close()

I think the problem is in the last part where the csvWriter is, maybe I put to much text in one line? As I said before I'm new at this and need tons of help.

Answer 1

I think the easiest solution utilizes pandas (which, interestingly, you imported but didn't use).

A working solution may look something like this:

import tweepy
import pandas as pd
consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth,wait_on_rate_limit=True)

# create list to append tweets to
tweets = []

# append all tweet data to list
for tweet in tweepy.Cursor(api.search,q="petya",count=100,
                       since="2017-04-03").items():
    tweets.append(tweet)

# convert 'tweets' list to pandas.DataFrame
tweets_df = pd.DataFrame(vars(tweets[i]) for i in range(len(tweets)))

# define file path (string) to save csv file to
FILE_PATH = </path/to/file.csv>

# use pandas to save dataframe to csv
tweets_df.to_csv(FILE_PATH)

And boom, you're done!

Note, if you only want to select a particular set of tweets, you can just create a list and then subset the dataframe afterwards.

e.g. (after the step where you convert the tweets to a pandas.DataFrame):

# define attributes you want
tweet_atts = [
'text', 'created_at', 'favorite_count',
'lang', 'retweet_count', 'source',
'in_reply_to_user_id_str', 'retweeted',
'id'
]

# subset dataframe
tweets_df = tweets_df[tweets_atts]

# save resulting df to csv
tweets_df.to_csv(FILE_PATH)

Feel free to reply if you need any more help!