Search code examples
pythonpython-3.xunixweb-scrapingpyppeteer

Why does this pyppeteer code only work on windows?


I have written a small program in python using pyppeteer. It runs fine on my Windows computer, but when I tried running it on a Unix-based system it did not work. Here's a minimal reproducible example:

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch(headless=True)
    page = await browser.newPage()
    await page.goto('http://www.example.com')
    print(await page.content(), flush=True)

asyncio.get_event_loop().run_until_complete(main())

On Windows it runs as expected and returns a HTML page. On Unix, I get this error after about 35 seconds of nothing happening:

Traceback (most recent call last):
  File "pyppeteerTest.py", line 10, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "pyppeteerTest.py", line 5, in main
    browser = await launch(headless=True)
  File "/home/thatcoolcoder/.local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 306, in launch
    return await Launcher(options, **kwargs).launch()
  File "/home/thatcoolcoder/.local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 167, in launch
    self.browserWSEndpoint = get_ws_endpoint(self.url)
  File "/home/thatcoolcoder/.local/lib/python3.6/site-packages/pyppeteer/launcher.py", line 226, in get_ws_endpoint
    raise BrowserError('Browser closed unexpectedly:\n')
pyppeteer.errors.BrowserError: Browser closed unexpectedly:

I am running python 3.8.2 on Windows and python 3.6.9 on Unix but I doubt that is the problem. I've tried the fixes suggested in the comments on this answer but that didn't change anything.


Solution

  • I found that some of pyppeteer's dependencies were missing from my Unix server. To fix it, I ran this command:

    sudo apt install gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget libcairo-gobject2 libxinerama1 libgtk2.0-0 libpangoft2-1.0-0 libthai0 libpixman-1-0 libxcb-render0 libharfbuzz0b libdatrie1 libgraphite2-3 libgbm1