I'm trying tornado framework. But I found tornado frequently failed after 30 seconds in the stress test (using multi-mechanize). I use 10 threads in multi-mechanize and run for 100 seconds, around 500 requests / seconds. And it's around 15% failure ratio after 30 seconds. The whole test is about 100 seconds. From the statistics, I realize, the failure may due to timeout after 0.2 seconds. I searched for several ways to increase the timeout on web, but nothing works.
The below is my tornado code:
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous
def get(self):
self.write("Hello, world")
self.finish()
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8000)
tornado.ioloop.IOLoop.instance().start()
Here is my multi-mechanize test script:
import requests
import random
import time
class Transaction(object):
def __init__(self):
pass
def run(self):
r = requests.get('http://127.0.0.1:8000')
output = r.raw.read()
assert(r.status_code == 200)
return output
if __name__ == '__main__':
trans = Transaction()
trans.run()
print trans.custom_timers
The following is the error message I got from multimech-run
Traceback (most recent call last):
File "././test_scripts/v_user.py", line 12, in run
r = requests.get('http://127.0.0.1:8000')
File "/Library/Python/2.7/site-packages/requests/api.py", line 54, in get
return request('get', url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/safe_mode.py", line 37, in wrapped
return function(method, url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/api.py", line 42, in request
return s.request(method=method, url=url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 230, in request
r.send(prefetch=prefetch)
File "/Library/Python/2.7/site-packages/requests/models.py", line 603, in send
timeout=self.timeout,
File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 415, in urlopen
body=body, headers=headers)
File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 267, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 941, in request
self._send_request(method, url, body, headers)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 975, in _send_request
self.endheaders(body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 937, in endheaders
self._send_output(message_body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 797, in _send_output
self.send(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 759, in send
self.connect()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 740, in connect
self.timeout, self.source_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 49] Can't assign requested address
To answer your question of how to change the timeout for tornado, you need to modify tornado's bind_socket function to set the time out: http://www.tornadoweb.org/documentation/_modules/tornado/netutil.html
sock.setblocking(0)
sock.bind(sockaddr)
sock.listen(backlog)
sockets.append(sock)
Change the first line to sock.setblocking(1). According to the documentation: http://docs.python.org/library/socket.html#socket.socket.settimeout :
Setting a timeout of None disables timeouts on socket operations s.settimeout(None) is equivalent to s.setblocking(1).
However, as suggested in the comment, I think you should look at distributing the load.