Search code examples
pythonseleniumgoogle-chromeselenium-webdriverautomation

Selenium can't open a second page

I am using Selenium to open different pages of a site. Have tried multiple times but the browser does not open a second webpage after the initial GET call. Have tried on both Chrome and Safari. Here is my code:

driver = webdriver.Chrome()
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.set_page_load_timeout(30)
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

Here is the error I get for the second call:

enter image description here

The info from Network logs is Error 504, but I have verified that it works perfectly when done on another window of the browser, without automation

Solution

  • A bit of more information about your usecase would have helped to construct a more canonical answer. However I was able to access the Page 2 of justdial.com/Chennai/Hr-Consultancy-Services with a minimized code block as follows:

    • Code Block:

      from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")

    • Browser Snapshot:

    justdial

    But while sending multiple get() one after another:

    driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")

    It seems ChromeDriver initiated Chrome Browser gets detected and the following error is shown:

    An error occurred while processing your request.
Reference #97.e5732c31.1612205693.6fd2708

    Solution

    To avoid the detection you can add the following option:

    --disable-blink-features=AutomationControlled

    Example

    from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-2")
driver.get("https://www.justdial.com/Chennai/Hr-Consultancy-Services/nct-10258625/page-3")