First and foremost; thanks in advance for any and all feedback provided.
I'm building a web scraper using the Playwright Python API. There's an element on a page, related to an authentication flow, that only loads after waiting a significant amount of time. When clicked, it redirects the user several times until the authentication process is complete.
To reduce the waiting time, I thought of executing the JavaScript associated to the element with page.evaluate(), since the JS file is shipped as soon as the page is loaded. Like so:
with page.expect_navigation(wait_until="domcontentloaded", url=targetUrl):
page.evaluate(r"startAuthFlow('authFlow');")
The method works: it always redirects to the desired page. However, around 20% of times, the following error occurs:
Execution context was destroyed, most likely because of a navigation
In testing, I've tried adding timeouts before and after the call to page.evaluate(), which on the surface seems to reduce the amount of crashes, however, the process is still unstable and adding timeouts beats the purpose of the solution.
I thought the page.expect_navigation() context was exactly for that purpose. Is there something I am fundamentally not understanding?
expect_navigation : This method is deprecated as its inherently racy, please use page.wait_for_url() instead.
page.click("a.delayed-navigation") # clicking the link will indirectly cause a navigation
page.wait_for_url("**/target.html")
Source: https://playwright.dev/python/docs/api/class-page#page-wait-for-navigation