Python webdriver script OK from console but won't start as an Upstart service

Question

I need to run a Python script that uses Selenium webdriver:

import platform
from selenium import webdriver
from pyvirtualdisplay import Display

driver = None

if platform.system() == 'Linux':
    print("Initializing browser for chrome...")
    DISPLAY = Display(visible=0, size=(800, 600))
    DISPLAY.start()
    print("Started display")
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('--no-sandbox')
    driver = webdriver.Chrome('/usr/local/bin/chromedriver', chrome_options=chrome_options)
    print("Done init chrome")
    connected = True
else:
    try:
        print("Starting firefox...")
        driver = webdriver.Firefox()
        connected = True
    except:
        print("Could not connect through firefox")

if driver:
    print("driver ok")
    driver.quit()
    print("All ok")

The script runs OK from console:

sudo ~/environments/scrapers/bin/python test_webdriver.py
Initializing browser for chrome...
Started display
Done init chrome
driver ok
All ok

But gives a WebDriverException error if trying to run using an exec stanza with Upstart:

Initializing browser for chrome...
Started display
...
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.26.436382 (70eb799287ce4c2208441fc057053a5b07ceabac),platform=Linux 4.8.0-58-generic x86_64)

I've added possibly absent paths in the upstart script, like this

env PYTHON_HOME=/home/rsa-key-20161031/environments/scrapers
env PATH=/usr/local/bin:/usr/bin:$PYTHON_HOME:$PATH
env ENV=production
env DISPLAY=:10
chdir /home/user/project
console log
exec $PYTHON_HOME/bin/python test_webdriver.py

With no effect. Search for error doesn't give anything specific to this. Any insight on how to get this working is much appreciated.

Update: My current solution is to use Cron since it doesn't seem to have problems using Xvfb. I'd still very much like to know if it is possible to run a webdriver task as a service. I also tried using Selenium as remote webdriver, with same negative results (Chrome exits after apparently not being able to connect to a virtual display)

Answer 1

I was confronted with a similar situation, while I configured pytest to run with selenium and firefox webdriver.

As a solution you could install xvfb i.e. in debian

sudo apt install xvfb

Now you could adjust your upstart file after the exec position, so that xvfb is called with your script as a parameter

xvfb-run --server-num=10 <script>

This way xvfb is started in front of your script.