Search code examples
progress-barpython-3.5python-asyncioaiohttptqdm

asyncio aiohttp progress bar with tqdm


I'm attempting to integrate a tqdm progress bar to monitor POST requests generated with aiohttp in Python 3.5. I have a working progress bar but can't seem to gather results using as_completed(). Pointers gratefully received.

Examples I've found suggest using the following pattern, which is incompatible with Python 3.5 async def definitions:

for f in tqdm.tqdm(asyncio.as_completed(tasks), total=len(coros)):
    yield from f

Working (albeit redacted) async code without the progress bar:

def async_classify(records):

    async def fetch(session, name, sequence):
        url = 'https://app.example.com/api/v0/search'
        payload = {'sequence': str(sequence)}
        async with session.post(url, data=payload) as response:
            return name, await response.json()

    async def loop():
        auth = aiohttp.BasicAuth(api_key)
        conn = aiohttp.TCPConnector(limit=100)
        with aiohttp.ClientSession(auth=auth, connector=conn) as session:
            tasks = [fetch(session, record.id, record.seq) for record in records]
            responses = await asyncio.gather(*tasks)    
        return OrderedDict(responses)

This is my unsuccessful attempt at modifying loop():

async def loop():
    auth = aiohttp.BasicAuth(api_key)
    conn = aiohttp.TCPConnector(limit=100)
    with aiohttp.ClientSession(auth=auth, connector=conn) as session:
        tasks = [fetch(session, record.id, record.seq) for record in records]
        for f in tqdm.tqdm(asyncio.as_completed(tasks), total=len(tasks)):
            await f
        responses = await asyncio.gather(f)
        print(responses)

Solution

  • await f returns a single response. Why would you pass an already completed Future to asyncio.gather(f) is unclear.

    Try:

    responses = []
    for f in tqdm.tqdm(asyncio.as_completed(tasks), total=len(tasks)):
        responses.append(await f)
    

    Python 3.6 implements PEP 530 -- Asynchronous Comprehensions:

    responses = [await f
                 for f in tqdm.tqdm(asyncio.as_completed(tasks), total=len(tasks))]
    

    It works inside async def functions now.