I have the following code.
#example.py
import asyncio
import aiohttp
async def foo():
conn = aiohttp.ProxyConnector(proxy='http://222.222.251.185:9797')
r = await aiohttp.get('https://www.python.org/', connector=conn)
return await r.read()
async def main():
tasks = []
for i in range(500):
tasks.append(foo())
for r in asyncio.as_completed(tasks):
print(await r)
parsed_data = asyncio.get_event_loop().run_until_complete(main())
When I do not use proxy, everything is OK. But when I use proxy, program is crashing with that trace:
Traceback (most recent call last):
File "/home/vika/Work/example.py", line 19, in <module>
parsed_data = asyncio.get_event_loop().run_until_complete(main())
File "/usr/lib/python3.5/asyncio/base_events.py", line 337, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/vika/Work/example.py", line 16, in main
print(await r)
File "/usr/lib/python3.5/asyncio/tasks.py", line 486, in _wait_for_one
return f.result() # May raise f.exception().
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/vika/Work/example.py", line 7, in foo
r = await aiohttp.get('https://www.python.org/', connector=conn)
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/client.py", line 552, in __await__
return (yield from self._coro)
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/client.py", line 174, in _request
conn = yield from self._connector.connect(req)
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/connector.py", line 307, in connect
transport, proto = yield from self._create_connection(req)
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/connector.py", line 683, in _create_connection
resp = yield from proxy_resp.start(conn, True)
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 597, in start
message = yield from httpstream.read()
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/streams.py", line 578, in read
result = yield from super().read()
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/streams.py", line 433, in read
yield from self._waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 358, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
aiohttp.errors.ServerDisconnectedError
Exception ignored in: <coroutine object foo at 0x7fcf51566518>
Traceback (most recent call last):
File "/home/vika/Work/example.py", line 7, in foo
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/client.py", line 552, in __await__
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/client.py", line 174, in _request
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/connector.py", line 307, in connect
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/connector.py", line 686, in _create_connection
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/connector.py", line 84, in close
File "/opt/python_custom/lib/python3.5/site-packages/aiohttp/connector.py", line 375, in _release
File "/usr/lib/python3.5/asyncio/selector_events.py", line 569, in close
File "/usr/lib/python3.5/asyncio/base_events.py", line 447, in call_soon
File "/usr/lib/python3.5/asyncio/base_events.py", line 456, in _call_soon
File "/usr/lib/python3.5/asyncio/base_events.py", line 284, in _check_closed
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object foo at 0x7fcf515664c0>
...
Task was destroyed but it is pending!
task: <Task pending coro=<foo() done, defined at /home/vika/Work/example.py:5> wait_for=<Future pending cb=[Task._wakeup()]> cb=[as_completed.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:472]>
Task was destroyed but it is pending!
task: <Task pending coro=<foo() done, defined at /home/vika/Work/example.py:5> wait_for=<Future pending cb=[Task._wakeup()]> cb=[as_completed.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:472]>
Task was destroyed but it is pending!
task: <Task pending coro=<foo() done, defined at /home/vika/Work/example.py:5> wait_for=<Future pending cb=[Task._wakeup()]> cb=[as_completed.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:472]>
Task was destroyed but it is pending!
task: <Task pending coro=<foo() done, defined at /home/vika/Work/example.py:5> wait_for=<Future pending cb=[Task._wakeup()]> cb=[as_completed.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:472]>
What could be the problem? I used different proxies, so th problem must be in code logic.
The proxy was wrong. Different proxies cause different errors, so it was hard to find one fine proxy. The code above is absolutely valid (but pls change the proxy!).