Sample code:
socket.setdefaulttimeout(150)
MechBrowser = mechanize.Browser()
Header = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)'}
Url = "http://example.com"
Data = "Justatest=whatever&letstry=doit"
Request = urllib2.Request(Url, Data, Header)
Response = MechBrowser.open(Request)
Response.close()
I don't think there's anything wrong with above codes, but every now and then I'll get hanging http POST request which prevents the whole script from continuously running. I already used socket.setdefaulttimeout(150)
how come it's not working? What is the reason causing this problem? And what should I do to get rid of this?
Found the problem.
I've been using requests
a lot recently and realized that the timeout
you set in both mechanize
and requests
are "NOT a time limit on the entire response download", which means if the connection is really slow and there are still data transferring, it will not timeout, which could hang the connection for quite a long time.
So what I've done is wrap those requests with threads and set timeout for those threads, and this way timeout is more accurate, just make sure you clean up/garbage collect those hanging connections.