When trying to use PlaywrightCrawler package in my server environment I get below error log.
[32mINFO[39m [33m PlaywrightCrawler:[39m Starting the crawl
[33mWARN[39m [33m PlaywrightCrawler:[39m Reclaiming failed request back to the list or queue. browserType.launchPersistentContext: Failed to launch: Error: spawn /root/.cache/ms-playwright/chromium-1033/chrome-linux/chrome ENOENT
============================================================
at async PlaywrightPlugin._launch (/usr/src/app/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)
at async BrowserPool._launchBrowser (/usr/src/app/node_modules/@crawlee/browser-pool/browser-pool.js:465:29)
at async /usr/src/app/node_modules/@crawlee/browser-pool/browser-pool.js:274:37[90m {"id":"erPiwuCRDH5Dzyr","url":"https://www.google.com.au/search?tbm=shop&hl=en-AU&psb=1&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL&q=Coles+Extra Virgin Avocado Oil 250mL&oq=Coles+Extra Virgin Avocado Oil 250mL&gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&sclient=products-cc&ts=658","method":"GET","uniqueKey":"https://www.google.com.au/search?gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&hl=en-AU&oq=Coles+Extra Virgin Avocado Oil 250mL&psb=1&q=Coles+Extra Virgin Avocado Oil 250mL&sclient=products-cc&tbm=shop&ts=658&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL"}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m All the requests from request list and/or request queue have been processed, the crawler will shut down.
[32mINFO[39m [33m PlaywrightCrawler:[39m Crawl finished. Final request statistics:[90m {"requestsFinished":0,"requestsFailed":1,"retryHistogram":[null,null,null,1],"requestAvgFailedDurationMillis":107,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":5,"requestTotalDurationMillis":107,"requestsTotal":1,"crawlerRuntimeMillis":11078}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m Error analysis:[90m {"totalErrors":1,"uniqueErrors":1,"mostCommonErrors":["1x: browserType.launchPersistentContext: Failed to launch: Error: spawn /root/.cache/ms-playwright/chromium-1033/chrome-linux/chrome ENOENT (/usr/src/app/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)"]}[39m
Here is my .ebextension/chromium.config:
packages:
yum:
cups-libs: []
dbus-glib: []
libXrandr: []
libXcursor: []
libXinerama: []
cairo: []
cairo-gobject: []
pango: []
libwayland-client: []
libwayland-cursor: []
gtk3: []
gdk-pixbuf2: []
libxkbcommon: []
libXScrnSaver: []
GConf2: []
atk: []
at-spi2-atk: []
at-spi2-core: []
alsa-lib: []
container_commands:
install_playwright:
command: "npx playwright install --with-deps chromium"
Below is the output of my yum.log:
Dec 08 10:06:18 Installed: libwayland-client-1.17.0-1.amzn2.x86_64
Dec 08 10:06:18 Installed: mesa-libglapi-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:18 Installed: atk-2.22.0-3.amzn2.0.2.x86_64
Dec 08 10:06:18 Installed: fontpackages-filesystem-1.44-8.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-fonts-common-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: libusbx-1.0.21-1.amzn2.x86_64
Dec 08 10:06:19 Installed: pixman-0.34.0-1.amzn2.0.2.x86_64
Dec 08 10:06:19 Installed: libxshmfence-1.2-1.amzn2.0.2.x86_64
Dec 08 10:06:19 Installed: 1:libglvnd-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
Dec 08 10:06:19 Installed: libwayland-server-1.17.0-1.amzn2.x86_64
Dec 08 10:06:19 Installed: mesa-libgbm-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:19 Installed: libgusb-0.2.9-1.amzn2.0.2.x86_64
Dec 08 10:06:19 Installed: 1:liberation-mono-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-serif-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-narrow-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-sans-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: 1:liberation-fonts-1.07.2-16.amzn2.noarch
Dec 08 10:06:19 Installed: dejavu-fonts-common-2.33-6.amzn2.noarch
Dec 08 10:06:19 Installed: dejavu-sans-fonts-2.33-6.amzn2.noarch
Dec 08 10:06:20 Installed: fontconfig-2.13.0-4.3.amzn2.x86_64
Dec 08 10:06:21 Installed: libwayland-cursor-1.17.0-1.amzn2.x86_64
Dec 08 10:06:21 Installed: graphite2-1.3.10-1.amzn2.0.2.x86_64
Dec 08 10:06:21 Installed: harfbuzz-1.7.5-2.amzn2.x86_64
Dec 08 10:06:21 Installed: vulkan-filesystem-1.0.61.1-2.amzn2.noarch
Dec 08 10:06:21 Installed: jasper-libs-1.900.1-33.amzn2.x86_64
Dec 08 10:06:21 Installed: alsa-lib-1.1.4.1-2.amzn2.x86_64
Dec 08 10:06:21 Installed: fribidi-1.0.2-1.amzn2.1.x86_64
Dec 08 10:06:21 Installed: libepoxy-1.3.1-2.amzn2.x86_64
Dec 08 10:06:21 Installed: json-glib-1.4.2-2.amzn2.x86_64
Dec 08 10:06:21 Installed: hicolor-icon-theme-0.12-7.amzn2.noarch
Dec 08 10:06:21 Installed: xkeyboard-config-2.20-1.amzn2.noarch
Dec 08 10:06:21 Installed: libxkbcommon-0.7.1-3.amzn2.x86_64
Dec 08 10:06:22 Installed: dconf-0.28.0-4.amzn2.x86_64
Dec 08 10:06:22 Installed: gsettings-desktop-schemas-3.28.0-3.amzn2.0.1.x86_64
Dec 08 10:06:22 Installed: avahi-libs-0.6.31-20.amzn2.x86_64
Dec 08 10:06:22 Installed: 1:cups-libs-1.6.3-51.amzn2.x86_64
Dec 08 10:06:22 Installed: libthai-0.1.14-9.amzn2.0.2.x86_64
Dec 08 10:06:22 Installed: trousers-0.3.14-2.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: gnutls-3.3.29-9.amzn2.0.1.x86_64
Dec 08 10:06:23 Installed: libXau-1.0.8-2.1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libxcb-1.12-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: vulkan-1.0.61.1-2.amzn2.x86_64
Dec 08 10:06:23 Installed: libX11-common-1.6.7-3.amzn2.0.2.noarch
Dec 08 10:06:23 Installed: libX11-1.6.7-3.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXext-1.3.3-3.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXrender-0.9.10-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXfixes-5.0.3-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXdamage-1.1.4-4.1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXrandr-1.5.1-2.amzn2.0.3.x86_64
Dec 08 10:06:23 Installed: libXi-1.7.9-1.amzn2.0.2.x86_64
Dec 08 10:06:23 Installed: libXcomposite-0.4.4-4.1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: gdk-pixbuf2-2.36.12-3.amzn2.x86_64
Dec 08 10:06:24 Installed: gtk-update-icon-cache-3.22.30-3.amzn2.x86_64
Dec 08 10:06:24 Installed: libXtst-1.2.3-1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: at-spi2-core-2.22.0-1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: at-spi2-atk-2.22.0-2.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: libXcursor-1.1.15-1.amzn2.x86_64
Dec 08 10:06:24 Installed: libXft-2.3.2-2.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: libXinerama-1.1.3-2.1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: libXxf86vm-1.1.4-1.amzn2.0.2.x86_64
Dec 08 10:06:24 Installed: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: 1:libglvnd-egl-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: mesa-libEGL-18.3.4-5.amzn2.0.1.x86_64
Dec 08 10:06:24 Installed: cairo-1.15.12-4.amzn2.x86_64
Dec 08 10:06:24 Installed: pango-1.42.4-4.amzn2.x86_64
Dec 08 10:06:24 Installed: cairo-gobject-1.15.12-4.amzn2.x86_64
Dec 08 10:06:24 Installed: libwayland-egl-1.17.0-1.amzn2.x86_64
Dec 08 10:06:24 Installed: 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch
Dec 08 10:06:24 Installed: desktop-file-utils-0.23-2.amzn2.x86_64
Dec 08 10:06:25 Installed: xdg-utils-1.1.0-0.17.20120809git.amzn2.noarch
Dec 08 10:06:25 Installed: lcms2-2.6-3.amzn2.0.2.x86_64
Dec 08 10:06:25 Installed: colord-libs-1.3.4-1.amzn2.0.2.x86_64
Dec 08 10:06:25 Installed: adwaita-cursor-theme-3.26.0-1.amzn2.noarch
Dec 08 10:06:28 Installed: adwaita-icon-theme-3.26.0-1.amzn2.noarch
Dec 08 10:06:28 Installed: libmodman-2.0.1-8.amzn2.0.2.x86_64
Dec 08 10:06:28 Installed: libproxy-0.4.11-10.amzn2.0.3.x86_64
Dec 08 10:06:28 Installed: glib-networking-2.56.1-1.amzn2.x86_64
Dec 08 10:06:28 Installed: libsoup-2.56.0-6.amzn2.x86_64
Dec 08 10:06:28 Installed: rest-0.8.0-2.amzn2.x86_64
Dec 08 10:06:29 Installed: gtk3-3.22.30-3.amzn2.x86_64
Dec 08 10:06:43 Installed: google-chrome-stable-108.0.5359.98-1.x86_64
Even all the chromium dependencies has been installed, still getting the issue.
There is no such issue when running it locally.
Hope someone can advise what is the issue.
UPDATE:
After further research found that the issue could be with the Docker image. So I replaced my existing Docker image with [zenika/alpine-chrome:with-playwright][1] which has playwright and headless chrome and included a new RUN command for npx playwright install. Now getting a new error which is:
[31mERROR[39m[33m PlaywrightCrawler:[39m Request failed and reached maximum retries. browserType.launchPersistentContext: Executable doesn't exist at /home/chrome/.cache/ms-playwright/chromium-1028/chrome-linux/chrome
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just installed or updated. ║
║ Please run the following command to download new browsers: ║
║ ║
║ npx playwright install ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════════════════════════════════════╝
at async PlaywrightPlugin._launch (/usr/src/app/node_modules/crawlee/node_modules/@crawlee/playwright/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)
at async BrowserPool._launchBrowser (/usr/src/app/node_modules/@crawlee/browser/node_modules/@crawlee/browser-pool/browser-pool.js:465:29)
at async /usr/src/app/node_modules/@crawlee/browser/node_modules/@crawlee/browser-pool/browser-pool.js:274:37[90m {"id":"tGOZOvUIBmhYpXW","url":"https://www.google.com.au/search?tbm=shop&hl=en-AU&psb=1&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL&q=Coles+Extra Virgin Avocado Oil 250mL&oq=Coles+Extra Virgin Avocado Oil 250mL&gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&sclient=products-cc&ts=782","method":"GET","uniqueKey":"https://www.google.com.au/search?gs_lcp=Cgtwcm9kdWN0cy1jYxADUABYAGAAaABwAHgAgAEAiAEAkgEAmAEA&hl=en-AU&oq=Coles+Extra Virgin Avocado Oil 250mL&psb=1&q=Coles+Extra Virgin Avocado Oil 250mL&sclient=products-cc&tbm=shop&ts=782&ved=2ahUKEwjP49q7gdb7AhUyn0sFHcYhAAQQu-kFegQIABAL"}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m All the requests from request list and/or request queue have been processed, the crawler will shut down.
[32mINFO[39m [33m PlaywrightCrawler:[39m Crawl finished. Final request statistics:[90m {"requestsFinished":0,"requestsFailed":1,"retryHistogram":[null,null,null,1],"requestAvgFailedDurationMillis":18,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":5,"requestTotalDurationMillis":18,"requestsTotal":1,"crawlerRuntimeMillis":11089}[39m
[32mINFO[39m [33m PlaywrightCrawler:[39m Error analysis:[90m {"totalErrors":1,"uniqueErrors":1,"mostCommonErrors":["1x: browserType.launchPersistentContext: Executable doesn't exist at /home/chrome/.cache/ms-playwright/chromium-1028/chrome-linux/chrome (/usr/src/app/node_modules/crawlee/node_modules/@crawlee/playwright/node_modules/@crawlee/browser-pool/playwright/playwright-plugin.js:99:40)"]}[39m
Here is my Dockerfile:
# # This file is a template, and might need editing before it works on your project.
FROM zenika/alpine-chrome:with-playwright
WORKDIR /usr/src/app
ENV PORT 3000
COPY package.json /usr/src/app/
RUN npm install --force
RUN npx playwright install
COPY . /usr/src/app
# replace this with your application's default port
EXPOSE 3000
CMD [ "npm", "start" ]
ALL FIXED.
Here is the Dockerfile
# Get the base image of Node version 16
FROM node:16
# Get the latest version of Playwright
FROM mcr.microsoft.com/playwright:focal
# Set the work directory for the application
WORKDIR /app
# COPY the needed files to the app folder in Docker image
COPY package*.json /app/
# Get the needed libraries to run Playwright
RUN apt-get update && apt-get -y install libnss3 libatk-bridge2.0-0 libdrm-dev libxkbcommon-dev libgbm-dev libasound-dev libatspi2.0-0 libxshmfence-dev
# Install the dependencies in Node environment
RUN npm ci
# Start the main script
CMD ["node", "--inspect=0.0.0.0:9229", "src/main.js"]
Here is the EB config
container:
install_playwright:
command: PLAYWRIGHT_BROWSERS_PATH=$HOME/pw-browsers npx playwright install