Search code examples
pythontornadopython-asyncio

error occurs when using asycio loop in tornado under high concurrency


I installed the asyncio Ioloop in tornado project, it ran with no problem with a single request, but when I did server stress test:

ab -n 10000 -c 4000  -p '/home/mwh/ad_tornado/for_test/task.json' -T 'application/x-www-form-urlencoded' '192.168.1.170/hwSdk/tsk/get_t_k.json'

It happened errors:

[E 161216 10:32:22 base_events:1090] Exception in callback BaseAsyncIOLoop._handle_events(11, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(11, 1)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/asyncio/events.py", line 125, in _run
  File "/home/mwh/virtualenv_mwh/lib/python3.4/site-packages/tornado/platform/asyncio.py", line 114, in _handle_events
  File "/home/mwh/virtualenv_mwh/lib/python3.4/site-packages/tornado/stack_context.py", line 275, in null_wrapper
  File "/home/mwh/virtualenv_mwh/lib/python3.4/site-packages/tornado/netutil.py", line 260, in accept_handler
  File "/usr/local/lib/python3.4/socket.py", line 187, in accept
OSError: [Errno 24] Too many open files

Is that use with asynico ioloop in tornado will degrade the performance?


Solution

  • You are trying to run 4000 parallel requests, which means 4000 opened sockets at the same time.

    But usually system limits allows only 1024 opened files by default (try ulimit -a).

    You could override the limit by ulimit -n 4096 call.