Search code examples
pythondatabasesqlitetwittertweepy

Stream tweets from Twitter (With Tweepy and Python) and store in SQLite database until N tweets has been processed


I a trying to store N tweets (filtered by the keyword "Trump") in a SQLite database table ("sentiment"). For this purpose, I have written a function "count_rows" that returns the integer number of rows stored in this table (each row corresponds to a tweet). Using this integer amount as a cutoff for reaching exactly N tweets I am attempting to stream tweets inside a while loop (see below):

N = 100

def count_rows():
    count_SQL = """SELECT COUNT(*) FROM sentiment"""
    c.execute(count_SQL)
    return c.fetchall()[0][0]

num_rows = count_rows()  # Starts at value: zero
while num_rows <= N:
    try:
        auth = OAuthHandler(ckey, csecret)
        auth.set_access_token(atoken, asecret)
        twitterStream = Stream(auth, listener())
        twitterStream.filter(track=["Trump"])
        # num_rows = count_rows()
    except Exception as e:
        print(str(e))
        time.sleep(5)
    num_rows = count_rows()

The challenge is, that the streaming goes on forever, and the while loop is infinite. What am I doing wrong? Am I using the count_rows() in a wrong way? (the function count_rows() itself works as it should, so the error must be within my while-loop logic I suppose).


Solution

  • [Note to self: remember to read all the words in a post :]

    EDIT:

    From the tweepy doc:

    Streams [do] not terminate unless the connection is closed,

    Shouldn't the count test be inside the listener? It sounds like the call to Stream is not going to return to the while function.


    Does this num_rows = count_rows() continuously reset num_rows to 0? There is no indication in the posted code that the COUNT(*) in sentiments changes. Are the tweets being inserted into sentiment table?