Search code examples
pythonpython-asyncioaiohttp

Closing aiohttp ClientSession when used in class


I have the following code

session = aiohttp.ClientSession()


async def fetch(session, url):
    while True:
        try:
            async with session.get(url) as response:
                assert response.status == 200
                return await response.json()
        except Exception as error:
            print(error)



class FPL():
    async def get_player_summaries(self, player_ids=[], return_json=False):
        tasks = [asyncio.ensure_future(
                fetch(session, API_URLS["player"].format(player_id)))
                for player_id in player_ids]

        player_summaries = await asyncio.gather(*tasks)

        if return_json:
            return player_summaries

        return [PlayerSummary(player_summary)
                for player_summary in player_summaries]

    async def get_points_against(self):
        players = await self.get_players(return_json=True)
        player_ids = [player["id"] for player in players]
        player_summaries = await self.get_player_summaries(
            player_ids, return_json=True)
        points_against = {}

        for player in players:
            try:
                player_summary = next(
                    summary for summary in player_summaries
                    if summary["history"][0]["element"] == player["id"])
            except Exception:
                continue
            player.update(player_summary)
            position = position_converter(player["element_type"]).lower()

            for fixture in player["history"]:
                if fixture["minutes"] == 0:
                    continue

                points = fixture["total_points"]
                opponent = team_converter(fixture["opponent_team"])
                location = "H" if fixture["was_home"] else "A"

                points_against.setdefault(
                    opponent,
                    {
                        "all": {"H": [], "A": []},
                        "goalkeeper": {"H": [], "A": []},
                        "defender": {"H": [], "A": []},
                        "midfielder": {"H": [], "A": []},
                        "forward": {"H": [], "A": []}
                    }
                )

                points_against[opponent]["all"][location].append(points)
                points_against[opponent][position][location].append(points)

        return points_against

which is used like so

from fpl import FPL
import asyncio

loop = asyncio.get_event_loop()
fpl = FPL()

x = loop.run_until_complete(fpl.get_points_against())
print(x)

this works fine and x is printed, but it's accompanied by the following warning

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fa2138111d0>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffac48>, 7377.304262674), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8ee8>, 7377.305252688), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8fa8>, 7377.306179118), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9528>, 7377.306921374), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc91c8>, 7377.307587193), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9d08>, 7377.308546534), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030be8>, 7377.324784641), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8b88>, 7377.325538414), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa4c8>, 7377.326132111), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc92e8>, 7377.327035234), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030fa8>, 7377.327984252), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8408>, 7377.328409375), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa5e8>, 7377.329149849), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8a08>, 7377.329776115), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa1c8>, 7377.330400698), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3708>, 7377.331131844), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8be8>, 7377.331974929), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3e88>, 7377.332880665), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3fa8>, 7377.339996297), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa168>, 7377.34077951), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3768>, 7377.341734826), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3dc8>, 7377.342455072), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9288>, 7377.343152698), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa228>, 7377.343844791), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3c48>, 7377.344872661), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3d08>, 7377.34534166), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffaee8>, 7377.346136623), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3408>, 7377.346773908), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa828>, 7377.347680292), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8348>, 7377.34843671), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c0305e8>, 7377.349162388), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3f48>, 7377.349909575), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffab88>, 7377.350612879), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffaa68>, 7377.351261815), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa588>, 7377.352141721), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8168>, 7377.352916437), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c0307c8>, 7377.353702472), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa3a8>, 7377.354645898), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030708>, 7377.355418555), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd32e8>, 7377.356222696), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030d08>, 7377.356890307), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030828>, 7377.357748581), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc98e8>, 7377.358468552), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe89a8>, 7377.36431476), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c0309a8>, 7377.365148168), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030648>, 7377.365791136), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa528>, 7377.36643204), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030dc8>, 7377.367176097), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c0308e8>, 7377.367818125), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9708>, 7377.36853057), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030ac8>, 7377.369177849), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030e88>, 7377.369894806), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa048>, 7377.370589053), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8e28>, 7377.371427258), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9ee8>, 7377.37224736), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8588>, 7377.372942966), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3048>, 7377.373591994), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe86a8>, 7377.374399274), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa0a8>, 7377.374866638), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3288>, 7377.375751613), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8528>, 7377.376432785), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd39a8>, 7377.377390439), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c2fcd68>, 7377.381260399), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe83a8>, 7377.382196696), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030a68>, 7377.382944559), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c030ee8>, 7377.3833559), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8468>, 7377.383995208), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffabe8>, 7377.38498033), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9c48>, 7377.385649648), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9108>, 7377.38617506), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c52bfa8>, 7377.388227618), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c52bf48>, 7377.388934452), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8d08>, 7377.393284709), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8108>, 7377.394200951), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9ca8>, 7377.395036827), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffae28>, 7377.396141727), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9b28>, 7377.396889149), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8888>, 7377.397542457), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa768>, 7377.398182665), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffaca8>, 7377.398876879), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9be8>, 7377.399550454), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa6a8>, 7377.400299738), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8288>, 7377.401028687), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c35a648>, 7377.401664671), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9648>, 7377.402413862), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8828>, 7377.403071713), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3228>, 7377.407205263), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9d68>, 7377.408117483), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa288>, 7377.40877989), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20d570be8>, 7377.409435516), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc96a8>, 7377.409980892), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffa468>, 7377.410663022), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe81c8>, 7377.411396958), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9a68>, 7377.412164899), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fc9348>, 7377.412951192), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3468>, 7377.413625465), (<aiohttp.client_proto.ResponseHandler object at 0x7fa20c52bee8>, 7377.414277707), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207ffad68>, 7377.414928317), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fe8ac8>, 7377.415578122), (<aiohttp.client_proto.ResponseHandler object at 0x7fa207fd3b88>, 7377.415948294)]']
connector: <aiohttp.connector.TCPConnector object at 0x7fa20d72c710>

The FPL class has many other functions, all which seem to work fine, so I can only assume the session is closed for those (the sessions not being closed is probably caused my the high number of requests made (~570), while the others are < 30). I've tried changing the class by making the session an argument that is passed to the class, and then closing it after, but that doesn't seem to work either. In the documentation it is mentioned that session context manager usage is not mandatory, but await session.close() method should be called in this case. Since the way I'm using it in a class isn't working, it is probably not the correct way of instantiating the session.

How are you supposed use an aiohttp ClientSession in a class?


Solution

  • There's no difference if you're closing session from class or not. Close it with

    await session.close()
    

    somewhere in the end of get_points_against. If for some reason it's not working for you – please test it with minimal code possible and post it here. Super-simple thing like

    import asyncio
    
    from aiohttp import ClientSession
    
    session = ClientSession()
    
    async def fetch(session, url):
        async with session.get(url) as r:
            return (await r.read())[:100]
    
    
    class Getter:
        async def print_data(self):
            urls = ['https://stackoverflow.com', 'https://google.com']
            for result in await asyncio.gather(*[fetch(session, url) for url in urls]):
                print(result)
    
            await session.close()
    
    loop = asyncio.get_event_loop()
    g = Getter()
    loop.run_until_complete(g.print_data())
    

    works fine with await session.close() and warns with the same warning as yours – without.


    Side note: aiohttp highly discourages against creating session outside of coros.