Search code examples
pythonweb-scrapingwebautomationplaywrightplaywright-python

How do you open multiple pages asynchronously with Playwright Python?


I want to open multiple urls at once using Playwright for Python. But I am struggling to figure out how. This is from the async documentation:

async def main():
    async with async_playwright() as p:
        for browser_type in [p.chromium, p.firefox, p.webkit]:
            browser = await browser_type.launch()
            page = await browser.newPage()
            await page.goto("https://scrapingant.com/")
            await page.screenshot(path=f"scrapingant-{browser_type.name}.png")
            await browser.close()

asyncio.get_event_loop().run_until_complete(main())

This opens each browser_type sequentially. How would I go about it if I wanted to do it in parallel? And how would I go about it if I wanted to do something similar with a list of urls?

I tried doing this:

urls = [
    "https://scrapethissite.com/pages/ajax-javascript/#2015",
    "https://scrapethissite.com/pages/ajax-javascript/#2014",
]
async def main(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.newPage()
        await page.goto(url)
        await browser.close()

async def go_to_url():
    tasks = [main(url) for url in urls]
    await asyncio.wait(tasks)

go_to_url()

But that gave me the following error:

92: RuntimeWarning: coroutine 'go_to_url' was never awaited
  go_to_url()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Solution

  • I believe you need to call your go_to_url function using the same recipe:

    asyncio.get_event_loop().run_until_complete(go_to_url())