Search code examples
pythonseleniumweb-scrapingtextwebdriverwait

How to extract the text from the webelements using Selenium and Python

Code trials:

driver.get(url)
cards = driver.find_elements_by_class_name("job-cardstyle__JobCardComponent-sc-1mbmxes-0")
for card in cards:
    data = card.get_attribute('text')
    print(data)

    
driver.close()
driver.quit()

The "cards" is returning selenium webelement and I am not able to extract the text from it by for loop.

Solution

  • This works to an extent:

    driver.get("https://www.monster.com/jobs/search?q=Python-Developer&where=Las+Vegas%2C+NV&page=1")
WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[@data-test-id = 'svx-job-title']")))
jobs = driver.find_elements(By.XPATH, "//div[contains(@class, 'job-cardstyle__JobCardHeader')]")
all_jobs = [job.text for job in jobs]
print(all_jobs)

    WebdriverWait imports:

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

    Output:

    ['Software engineer III\nRandstad USA\nLas Vegas, NV', 'C\nPython Developer\nconfidential\n$55 - $65 / Per Hour', 'C\nSenior Software Engineer\nCox Communications Inc\nLas Vegas, NV', 'Mission Systems Engineer\nDCS Corporation\nLas Vegas, NV', 'G\nSoftware Engineer - 914\nGCR Technical Staffing\nHenderson, NV', 'Z\nNetSuite Developer\nZone & Company Software Consulting\nLas Vegas, NV', 'IT Project Engineer\nRauland Florida by Ametek, Inc.\nSunrise, NV', 'A\nWeb Developer\nArdor Global', 'Senior Software Engineer – Node\nMeridian Technology Group Inc.']

Process finished with exit code 0

    You may split the list with \n delimiter for further usage. Also, seems like this site loads the cards dynamically, i.e., as you scroll down, new cards load up, so you may not get all the cards in one instance.