Exception handling on asyncronously HTTP-requests in Python 3.x

Question

I'm trying to handle an asynchronous HTTP request. I call the async_provider() function from another module and with the resulting response.text() I perform subsequent tasks. It only works if all requests are successful. But I can't handle any exceptions for failed requests (whatever the reason for the exception). Thank you for your help. Here is the relevant part of the code:

import asyncio
import aiohttp

# i call this function from another module
def async_provider():
    list_a, list_b = asyncio.run(main())
    return list_a, list_b


async def fetch(session, url):
    # session.post request cases
    if url == "http://...1":
        referer = "http://...referer"
        user_agent = (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) "
            "AppleWebKit/605.1.15 (KHTML, like Gecko) "
            "Version/12.1 Safari/605.1.15"
        )
        payload = {'key1': 'value1', 'key2': 'value2'}
        async with session.post(
            url, data=payload, headers={"Referer": referer, "User-Agent": user_agent}
        ) as response:
            if response.status != 200:
                response.raise_for_status()
            return await response.text()

    # session.get request cases
    else:
        async with session.get(url) as response:
            if response.status != 200:
                response.raise_for_status()
            return await response.text()


async def fetch_all(session, urls):
    results = await asyncio.gather(
        *[asyncio.create_task(fetch(session, url)) for url in urls]
    )
    return results


async def main():
    urls = ["http://...1", "http://...2", "http://...3"]
    async with aiohttp.ClientSession() as session:
        response_text_1, response_text_2, response_text_3 = await fetch_all(
            session, urls
        )
    # some task with response text

Any exception breaks all requests

Answer 1

Check "return_exceptions" flag on gather.

results = await asyncio.gather(
    *[asyncio.create_task(fetch(session, url)) for url in urls],
    return_exceptions=True
)

It will return you list of finished tasks. You can then use their Task.result() or Task.exception() methods to reraise or check if there was exception.