Search code examples
pythonseleniumxpathwebdriverwaitwindow-handles

How to open each product within a website in a new tab for scraping using Selenium through Python

I am scraping a web site using selenium "https://www.medline.com/catalog/category-products.jsp?itemId=Z05-CA02_03&N=111079+4294770643&iclp=Z05-CA02_03"

For single page and single product i am able to scrape by passing the product url but i am trying to do so by selenium i.e auto selection of product an page after select all the product one by one and it should move to next page and after opening product details page it should scrape which is done by beautiful soup here is product url from the base url "https://www.medline.com/product/SensiCare-Powder-Free-Nitrile-Exam-Gloves/SensiCare/Z05-PF00342?question=&index=P1&indexCount=1"

Here is my code:

chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(executable_path='C:/Users/ptiwar34/Documents/chromedriver.exe', chrome_options=chromeOptions, desired_capabilities=chromeOptions.to_capabilities())
driver.get("https://www.medline.com/catalog/category-products.jsp?itemId=Z05-CA02_03&N=111079+4294770643&iclp=Z05-CA02_03")

while True:
    try:  
        WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(@class, 'resultGalleryViewRow')]//div[@class='medGridProdTitle']//a[contains(@href]"))).click()
        print("Clicked for next page")
    except TimeoutException:
        print("No more pages")
        break
driver.quit()

Here it does not throws error

It does not open page for each product , I want to open each product in new tab and after scraping it delete and open the new tab for a new product

Solution

  • From the webpage https://www.medline.com/catalog/category-products.jsp?itemId=Z05-CA02_03&N=111079+4294770643&iclp=Z05-CA02_03 to open each product in new tab and scrape it you have to induce WebDriverWait for the number_of_windows_to_be(2) and you can use the following Locator Strategies:

    • Code Block:

        from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  import time

  chrome_options = webdriver.ChromeOptions() 
  chrome_options.add_argument("start-maximized")
  driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\WebDrivers\chromedriver.exe')

  driver.get("https://www.medline.com/catalog/category-products.jsp?itemId=Z05-CA02_03&N=111079+4294770643&iclp=Z05-CA02_03")
  my_hrefs = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class, 'resultGalleryViewRow')]//div[@class='medGridProdTitle']//a")))]
  windows_before  = driver.current_window_handle # Store the parent_window_handle for future use
  for my_href in my_hrefs:
      driver.execute_script("window.open('" + my_href +"');")
      WebDriverWait(driver, 10).until(EC.number_of_windows_to_be(2)) # Induce  WebDriverWait for the number_of_windows_to_be 2
      windows_after = driver.window_handles
      new_window = [x for x in windows_after if x != windows_before][0] # Identify the newly opened window
      driver.switch_to.window(new_window) # switch_to the new window
      time.sleep(3) # perform your webscraping here
      print(driver.title) # print the page title or your perform your webscraping
      driver.close() # close the window
      driver.switch_to.window(windows_before) # switch_to the parent_window_handle
  driver.quit() #quit your program

    • Console Output:

        SensiCare Powder-Free Nitrile Exam Gloves | Medline Industries, Inc.
  MediGuard Vinyl Synthetic Exam Gloves | Medline Industries, Inc.
  CURAD Stretch Vinyl Exam Gloves | Medline Industries, Inc.
  CURAD Nitrile Exam Gloves | Medline Industries, Inc.
  SensiCare Ice Blue Powder-Free Nitrile Exam Gloves | Medline Industries, Inc.
  MediGuard Synthetic Exam Gloves | Medline Industries, Inc.
  Accutouch Synthetic Exam Gloves | Medline Industries, Inc.
  Aloetouch Ice Powder-Free Nitrile Exam Gloves | Medline Industries, Inc.
  Aloetouch 3G Powder-Free Synthetic Exam Gloves | Medline Industries, Inc.
  SensiCare Powder-Free Stretch Vinyl Sterile Exam Gloves | Medline Industries, Inc.
  CURAD Powder-Free Textured Latex Exam Gloves | Medline Industries, Inc.
  Accutouch Chemo Nitrile Exam Gloves | Medline Industries, Inc.
  Aloetouch 12" Powder-Free Nitrile Exam Gloves | Medline Industries, Inc.
  Ultra Stretch Synthetic Exam Gloves | Medline Industries, Inc.
  Generation Pink 3G Synthetic Exam Gloves | Medline Industries, Inc.
  SensiCare Extended Cuff Powder-Free Nitrile Exam Gloves | Medline Industries, Inc.
  Eudermic MP High-Risk Powder-Free Latex Exam Gloves | Medline Industries, Inc.
  Aloetouch Powder-Free Latex Exam Gloves | Medline Industries, Inc.
  CURAD Powder-Free Nitrile Exam Gloves | Medline Industries, Inc.
  Medline Sterile Powder-Free Latex Exam Gloves | Medline Industries, Inc.
  SensiCare Silk Powder-Free Nitrile Exam Gloves | Medline Industries, Inc.
  Medline Sterile Powder-Free Latex Exam Glove Pairs | Medline Industries, Inc.
  MediGuard 2.0 Nitrile Exam Gloves | Medline Industries, Inc.
  Designer Boxed Vinyl Exam Gloves | Medline Industries, Inc.

    References

    You can find a couple of relevant detailed discussions in: