Search code examples
python-3.xgoogle-chromeseleniumpm2

Running a Python app with Selenium and Chromedriver using PM2 constantly crashes and restarts


I've been having an issue getting my app to run in PM2. It's a simple Python 3 app that uses Selenium and Chrome/chromedriver to scrape our website. I think the problem is either that Chrome itself is located in a different folder outside of the apps folders or that it loses connection with Chrome/chromedriver process once it starts, and I just don't know how to tell PM2 to include those processes.

The script runs fine without PM2, so I'm not sure where the issue is or how to fix it. Here's the output that it generates after every restart (it basically starts, crashes and restarts constantly, without ever connecting to the website):

0|Scraper  | [2018-04-11 10:07:32.222861] STARTING SCRAPER
0|Scraper  | Traceback (most recent call last):
0|Scraper  |   File "/home/myapp/Scraper.py", line 1617, in <module>
0|Scraper  |     process()
0|Scraper  |   File "/home/myapp/Scraper.py", line 164, in process
0|Scraper  |     browser              = webdriver.Chrome(chrome_options = options, executable_path = path_to_chromedriver)
0|Scraper  |   File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/chrome/webdriver.py", line 62, in __init__
0|Scraper  |     self.service.start()
0|Scraper  |   File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/common/service.py", line 100, in start
0|Scraper  |     time.sleep(1)
0|Scraper  | KeyboardInterrupt  <--- I'm guessing this is PM2 killing the app after an error?
0|Scraper  |     Progress is Empty  <--- 'Empty' because it never did anything
0|Scraper  |     Progress SAVED

These are my chrome options:

options  = webdriver.ChromeOptions()
options.binary_location = '/usr/bin/google-chrome'
options.add_argument('headless')
options.add_argument('--log-level=3')
options.add_argument('window-size=1920x1080')
options.add_argument('--mute-audio')
options.add_argument('no-sandbox')

And the chromedriver itself is located directly in the apps parent directory.

Here's my PM2 process json file:

{
  "apps" : 
  [
    {
      "name"        : "Scraper",
      "script"      : "Scraper.py",
      "interpreter" : "python3",
      "watch"       : true
    }
  ]
}

All I do is PM2 start Scraper-process.json and then PM2 logs Scraper to view the output.

EDIT: Here's the initialization of the chromedriver

try:
    path_to_chromedriver = "chromedriver"
    browser              = webdriver.Chrome(chrome_options = options, executable_path = path_to_chromedriver)
except Exception as e:
    print("[ERROR] Chromedriver: " + str(e))

And then I start by sending it to our page:

browser.get(ourURL)

Solution

  • It seems the issue was in my PM2 process file. I mistakenly set "watch" to true, forgetting that the app makes changes to files often, which causes pm2 to restart the app every time.