I have been developing (on my Mac OSX) with Selenium Web Driver to do some scraping but I need to move the script to a Headless Linux Server.
With some research, it seems I need to use PyVirtualDisplay to simulate the launching of browser for Selenium when in a Headless Server. Below is my code and the error I am getting.
Code: (Source)
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()
Error I am getting:
My-MBP:learningpython neilnidhi$ python seleniumheadless.py
Traceback (most recent call last):
File "seleniumheadless.py", line 4, in <module>
display = Display(visible=0, size=(800, 600))
File "/usr/local/lib/python2.7/site- packages/pyvirtualdisplay/display.py", line 33, in __init__
self._obj = self.display_class(
File "/usr/local/lib/python2.7/site-packages/pyvirtualdisplay/display.py", line 51, in display_class
cls.check_installed()
File "/usr/local/lib/python2.7/site-packages/pyvirtualdisplay/xvfb.py", line 38, in check_installed
ubuntu_package=PACKAGE).check_installed()
File "/usr/local/lib/python2.7/site-packages/easyprocess/__init__.py", line 209, in check_installed
raise EasyProcessCheckInstalledError(self)
easyprocess.EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
OSError=[Errno 2] No such file or directory
Program install error!
The solution to this problem is very simple. You don't have an xserver runnning that pyvirtualdisplay could attach itself to. Try installing xvfb or any other x11 server with its fonts and utils and it should just work fine. I apparently have a script to install the entire xvfb stack on linux though.
But as @Thomas Orozco pointed out, PhantomJS does the job pretty well.