Search code examples
pythonnodriver

Following links across multiple pages

I am accessing a website that has multiple pages. You advance by highlighting/hovering over a drop down and then selecting one of the options that pops up. When you make the selection, you advance to a new page. Unfortunately when I get to the new page I don't seem to have the reference I need to keep going. When I try to access content on that new page I get an Event Loop is closed error. My code is as follows:

import asyncio
import datetime
import nodriver as uc
import pyautogui
import tracemalloc
from time import sleep
async def b_start_chrome_and_log_in():
    print("start chrome")
    # start a new Chrome instance
    driver = await uc.start(maximize=True)
    #   Force window to maximum size - crude method - fix someday
    pyautogui.click(x=1181, y=38)
    # visit the target website
    page = await driver.get('https://www.mytpi.com/')
    print("???")
    sleep(2)
    #   Select Improve my Game Dropdown
    #   Find Drop Down
    improve = await page.find('/html/body/nav/div/ul/li[4]/a')
    # Select Pay per Visit
    sleep(2)
    page2 = await improve.click()
    #   Find swing characteristics
    print("looking for improve")
    swing = await page2.find('/html/body/div[2]/div/div/ul/li[1]/div/a/span')

asyncio.run(b_start_chrome_and_log_in())

I get the following error messages:

    Traceback (most recent call last):
  File "C:\MyStuff\Travels\Python312\Projects\Automated Tee Times\NoDriver1.py", line 28, in <module>
    asyncio.run(b_start_chrome_and_log_in())
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\MyStuff\Travels\Python312\Projects\Automated Tee Times\NoDriver1.py", line 26, in b_start_chrome_and_log_in
    swing = await page2.find('/html/body/div[2]/div/div/ul/li[1]/div/a/span')
                  ^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'find'
successfully removed temp profile C:\Users\pinev\AppData\Local\Temp\uc_rc0vuuxa
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x000001D18A85B600>
Traceback (most recent call last):
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\base_subprocess.py", line 126, in __del__
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\base_subprocess.py", line 104, in close
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\proactor_events.py", line 109, in close
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\base_events.py", line 772, in call_soon
  File "C:\MyStuff\Travels\Python312\Lib\asyncio\base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

I am just guessing, but it seems like I need to store the details of the new page and then reference them in the next call. I tried that by putting the page2 reference in the line before the .find. The next to the last line (starts swing =) is where it blows up.

Solution

  • its look like the problem is how you're interacting with the webpage and handling transitions between pages. Let's break it down:

    The Main Problem

    The improve.click() line:

    In most asynchronous browser automation libraries, click() doesn't necessarily return a new page object. Instead, it just performs the click action. The new page or the updated content needs to be explicitly retrieved or waited for after the click. Since page2 is assigned as the result of improve.click(), it becomes None. NoneType Error:

    When you try to call .find() on page2, which is None, Python raises the AttributeError because None doesn't have a .find() method. Event Loop Error:

    The "Event loop is closed" error occurs because the program crashes, and the asyncio loop isn't properly shut down or cleaned up. Solution To resolve this issue, you need to:

    Properly wait for the page navigation to complete after the click. Avoid assuming click() returns the new page; instead, wait for the new page's content to load. Use reliable selectors to locate elements. Here's an updated version of your code:

    import asyncio
import nodriver as uc
import pyautogui
from time import sleep

async def b_start_chrome_and_log_in():
    print("start chrome")
    
    driver = await uc.start(maximize=True)
    
    pyautogui.click(x=1181, y=38)
    
    await driver.get('https://www.mytpi.com/')
    print("Website loaded")
    sleep(2)
    
    improve = await driver.find('/html/body/nav/div/ul/li[4]/a')
    await improve.hover()  # Hover over the dropdown
    sleep(1)
    
    pay_per_visit = await driver.find('/html/body/nav/div/ul/li[4]/ul/li[1]/a')
    await pay_per_visit.click()
    
    await driver.wait_for_navigation()
    print("Navigated to new page")
    
    swing = await driver.find('/html/body/div[2]/div/div/ul/li[1]/div/a/span')
    print("Found swing characteristics:", swing)

    await driver.close()

asyncio.run(b_start_chrome_and_log_in())