Tracemalloc Error while running HTMLrequests()

Question

I'm writing an Discord-Bot which sends an web-hook including infos scraped from an website.

The Function which scrapes the infos is the following:

import json
from requests_html import AsyncHTMLSession
import requests_html
import nest_asyncio
from bs4 import BeautifulSoup


class Product:

    def __init__(self, url):
        self._url = url

    def get_sizes_prices(self):
        nest_asyncio.apply()
        asession = AsyncHTMLSession()

        # create session from given endpoint with keywords/sku
        async def get_url():
            slug = asession.get('a special URL' + self._url)

        result = asession.run(get_url())
        result_slug = result.html

        **Some more code down there which gets some special things from the scraped website**
        return **variables**

While Running this, I receive the Error:

Ignoring exception in on_message
Traceback (most recent call last):
  File "C:\Users\lucab\PycharmProjects\untitled\venv\lib\site-packages\discord\client.py", line 333, in _run_event
    await coro(*args, **kwargs)
  File "C:/Users/lucab/OneDrive/Bots_-Coding/NexusTools/restocksscraper/main.py", line 15, in on_message
    array = product_obj.get_sizes_prices()
  File "C:\Users\lucab\OneDrive\Bots_-Coding\NexusTools\restocksscraper\get_product.py", line 21, in get_sizes_prices
    results = asession.run(getlink())
  File "C:\Users\lucab\PycharmProjects\untitled\venv\lib\site-packages\requests_html.py", line 772, in run
    asyncio.ensure_future(coro()) for coro in coros
  File "C:\Users\lucab\PycharmProjects\untitled\venv\lib\site-packages\requests_html.py", line 772, in <listcomp>
    asyncio.ensure_future(coro()) for coro in coros
TypeError: 'coroutine' object is not callable
C:\Users\lucab\PycharmProjects\untitled\venv\lib\site-packages\discord\client.py:340: RuntimeWarning: coroutine 'Product.get_sizes_prices.<locals>.getlink' was never awaited
  pass
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

firstI used just HTMLSession(), after checking the documentation, i use AsyncHTMLSession, bc this is recommended by requests_html(https://pypi.org/project/requests-html/)

Also my Discord-Bot-Code:

import discord , os
from dotenv import load_dotenv
from get_product import Product

load_dotenv()
TOKEN = os.getenv('DISCORD_TOKEN')
client = discord.Client()

@client.event
async def on_message(message):
    if message.content.startswith('!test'):
        product = message.content
        product.replace('!test', '') #to only get the string after the !test command
        product_obj = Product(product)
        array = product_obj.get_sizes_prices() #the return values are stored in "array"

        varembed = discord.Embed(
            title = "XXXXXX",
            description='SKU: '+XXXXX,
            url = XXXXXX,
            color = discord.Colour.darker_grey(),
        )
        varembed.set_footer(text="XXXX" )

        for i in array[0]:
            for x in array[1]:
                varembed.add_field(name="",value="", inline=True)

        await message.channel.send(embed=varembed)

client.run(TOKEN)

EDIT: What worked for me was: calling the function in the Discord-Bot with an await:

array = await product_obj.get_sizes_prices()

setting async to the function itself:

async def get_sizes_prices(self):

and setting the async_session on await:

result = await async_session.get('some url' + self._url)
result_slug = result.text

Answer 1

This error is usually caused by you not awaiting an async function. In this case, I believe it is caused when you invoke get_url().I assume you created that embedded function because it was asking you to await asession.get and you couldn't because get_sizes_prices wasn't async either.

Below is some untested code that should solve your issue.

async def get_sizes_prices(self):
    nest_asyncio.apply()
    asession = AsyncHTMLSession()

    # create session from given endpoint with keywords/sku
    response = await asession.get('a special URL' + self._url)

    result_slug = response.html

What I've done is removed the embedded function. Added the await statement to asession.get and made the entire function async. As a result, you will also need to await this function wherever you call it in the first place.

I used the documentation here as reference to what methods needed to be awaited or not. Hope that solves your issue :)