Search code examples
pythonseleniumsyntaxweb-crawlerempty-list

Get an empty List despite no error and assuming the logic is correct

im getting an empty list when i print the below, i'm pretty sure the code is correct

I am trying to retrieve the values for both class level-0 and level-1

website:https://stamprally.org/

prefectureValues = []
prefectureValueStorage = driver.find_elements_by_class_name(
    'div.header_search_inputs>select#header_search_cat1 > option.level-0.level-1')

for prefectureCode in prefectureValueStorage:
    prefectureValues.append(prefectureCode.get_attribute('value'))

print(prefectureValues)

Wanted List values

Solution 
  • //select[@name='search_cat1']/option[@class='level-1' or @class='level-0']

    Should be a simple xpath to use to get all the options with those classes.

    #header_search_cat1>:is(option.level-1 ,option.level-0)

    Would be the css selector for the above.

    So to pull it all together

    prefectureValues =[x.get_attribute('value') for x in driver.find_elements_by_xpath("//select[@name='search_cat1']/option[@class='level-1' or @class='level-0']")]
print(prefectureValues)

    Outputs

    ['145', '66', '67', '68', '69', '70', '71', '72', '73', '74', '75', '76', '77', '78', '79', '80', '81', '82', '83', '84', '85', '86', '87', '88', '89', '90', '91', '92', '94', '93', '95', '96', '97', '98', '99', '100', '101', '102', '103', '104', '105', '106', '107', '108', '109', '110', '111', '112', '113', '114', '115', '116', '117', '118', '119', '120']

    You could also use waits and etc

    from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
wait=WebDriverWait(driver, 10)
driver.get("https://stamprally.org/")
searchCat1Options=wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"#header_search_cat1>:is(option.level-1 ,option.level-0)")))
prefectureValues=[x.get_attribute('value') for x in searchCat1Options]
print(prefectureValues)