Search code examples
pythoncurltwitterpycurl

Handle pycurl hang on Twitter streaming api


I am using pycurl to connect to the twitter streaming API.

This works well but sometimes after running for a few hours it will stop hang indefinitely, not throwing any exceptions. How can I detect/handle a hang in this script?

import pycurl, json

STREAM_URL = "http://stream.twitter.com/1/statuses/filter.json"

USER = "presidentskroob"
PASS = "12345"

def on_receive(data):
  print data

conn = pycurl.Curl()
conn.setopt(pycurl.USERPWD, "%s:%s" % (USER, PASS))
conn.setopt(pycurl.URL, STREAM_URL)
conn.setopt(pycurl.WRITEFUNCTION, on_receive)
conn.perform()

Solution

  • FROM: http://man-wiki.net/index.php/3:curl_easy_setopt

    CURLOPT_LOW_SPEED_LIMIT - Pass a long as parameter. It contains the transfer speed in bytes per second that the transfer should be below during CURLOPT_LOW_SPEED_TIME seconds for the library to consider it too slow and abort.

    and

    CURLOPT_LOW_SPEED_TIME - Pass a long as parameter. It contains the time in seconds that the transfer should be below the CURLOPT_LOW_SPEED_LIMIT for the library to consider it too slow and abort.


    Example:

    conn.setopt(pycurl.LOW_SPEED_LIMIT, 1)
    conn.setopt(pycurl.LOW_SPEED_TIME, 90)