Search code examples
pythonseleniumselenium-webdriverfirefoxcron

Trying to run a python script through crontab but getting errors related to Remote webdriver init

Here is a python script. Running in Python 3.8. Browser- Firefox and geckodriver (latest versions)

#!/usr/bin/python3

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary (#added later to solve)

if __name__ == '__main__': (#added later to solve error)

binary = FirefoxBinary('/usr/bin/firefox') (#added later to solve error)
z=''
with open('dailynews.txt','w+') as fileobj: 
    dailynews=['State Bank of India']

for z in dailynews:
  
    wbpg='https://news.google.com/search?q=' + str(z) + ' when:1d'
    driver=webdriver.Firefox() (#This was earlier and worked fine.)        
    driver=webdriver.Firefox(firefox_binary=binary,executable_path='/usr/bin/geckodriver')
    (#added later to solve)
    driver.get(wbpg)
    
    articlelist=WebDriverWait(driver,20).until(EC.presence_of_all_elements_located
    ((By.XPATH,"//*@class='DY5T1d RZIKme']")))  
    articlelist=driver.find_elements_by_xpath("//*[@class='DY5T1d RZIKme']")
    
    for i in articlelist:
        print(i.text,file=fileobj) #,file=fileobj >> to write to a file
    driver.quit()

'''

This script works fine in python ide (eclipse>PyDev) and runs well through terminal as well. However, I scheduled the same in crontab (using crontab -e).

* * * * * /usr/bin/python3 /Documents/morning_newspaper.py >> ~/Documents/cron.log 2>&1)

The job is running but I am getting following errors:

Traceback (most recent call last): File "/home/leopard/Documents/morning_newspaper.py", line 22, in driver=webdriver.Firefox(firefox_binary=binary,executable_path='/usr/bin/geckodriver') File "/home/leopard/.local/lib/python3.8/site-packages/selenium/webdriver/firefox webdriver.py", line 170, in init RemoteWebDriver.init( File "/home/leopard/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in init self.start_session(capabilities, browser_profile) File "/home/leopard/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session response = self.execute(Command.NEW_SESSION, parameters) File "/home/leopard/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/home/leopard/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status 1

This is the syslog snippet from the job:

Feb 8 12:23:01 leopard CRON[7344]: (leopard) CMD (/usr/bin/python3 /Documents/morning_newspaper.py >> ~/Documents/cron.log 2>&1) Feb 8 12:23:01 leopard dbus-daemon[1552]: [session uid=1000 pid=1552] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=1548 comm="/usr/libexec/tracker-miner-fs " label="unconfined") Feb 8 12:23:01 leopard systemd[1540]: Starting Tracker metadata database store and lookup manager... Feb 8 12:23:01 leopard dbus-daemon[1552]: [session uid=1000 pid=1552] Successfully activated service 'org.freedesktop.Tracker1' Feb 8 12:23:01 leopard systemd[1540]: Started Tracker metadata database store and lookup manager. Feb 8 12:23:01 leopard dbus-daemon[1552]: [session uid=1000 pid=1552] Activating via systemd: service name='org.freedesktop.Tracker1.Miner.Extract' unit='tracker-extract.service' requested by ':1.1' (uid=1000 pid=1548 comm="/usr/libexec/tracker-miner-fs " label="unconfined") Feb 8 12:23:01 leopard systemd[1540]: Starting Tracker metadata extractor... Feb 8 12:23:01 leopard tracker-extract[7356]: Set scheduler policy to SCHED_IDLE Feb 8 12:23:01 leopard tracker-extract[7356]: Setting priority nice level to 19 Feb 8 12:23:01 leopard dbus-daemon[1552]: [session uid=1000 pid=1552] Successfully activated service 'org.freedesktop.Tracker1.Miner.Extract' Feb 8 12:23:01 leopard systemd[1540]: Started Tracker metadata extractor. Feb 8 12:23:12 leopard systemd[1540]: tracker-extract.service: Succeeded. Feb 8 12:23:31 leopard tracker-store[7349]: OK Feb 8 12:23:31 leopard systemd[1540]: tracker-store.service: Succeeded.

Initially, thought that this would be due to driver initiation so added Firefox binary and all (from a post on stackoverflow) but this does not work.

Thanks for your anticipated solutions. '''

Solution

  • The user/environment that cron is using to execute the task might not have the permission/ability to launch a UI process.

    To remedy this you can use headless mode:

    ...
binary = FirefoxBinary('/usr/bin/firefox')
binary.add_command_line_options('-headless')
...