Search code examples
pythonpython-asynciofreezeaiohttp

Can't convert aiohttp request to text while using rule34 api


I wanted to make a request to rule34 api using aiohttp, but when I use text() or json() method the program freezes:

import aiohttp
import asyncio

async def foo():
    async with aiohttp.ClientSession() as session:
        r = await session.get('https://api.rule34.xxx/index.php?page=dapi&s=post&q=index&json=1&limit=1000')
    print('got response') #ok
    text = await r.text() #freezes
    print(text) #never reaches

asyncio.run(foo())

But when I use limit=1 in URL it works normally.

Also it works with requests library with any limit value:

import requests
r = requests.get('https://api.rule34.xxx/index.php?page=dapi&s=post&q=index&json=1&limit=1000')
print(r.text)

I tried to use debugger and noticed that r._body was None before await r.text() was called and the data I need is located at r.content._buffer and looks like deque([b'data']), so I tried to use it, but then I noticed that the data isn't complete, like it isn't downloaded completely:

_buffer value.

Maybe because of the r wasn't awaited or I don't know, but I can't use await r because it throws an exception, or await r.text() because it freezes. So I can't even find the reason why it happens.


Solution

  • All of your interaction with aiohttp needs to be inside the ClientSession context manager, so you need (note the change in indentation):

    import aiohttp
    import asyncio
    
    async def foo():
        async with aiohttp.ClientSession() as session:
            r = await session.get('https://api.rule34.xxx/index.php?page=dapi&s=post&q=index&json=1&limit=1000')
            print('got response')
            text = await r.text()
            print(text)
    
    asyncio.run(foo())
    

    This successfully prints a JSON response from the API.