I am trying to scrape w dynamic jobs website that uses JS to load jobs after specifying your filters. I used BS4 and Selenuim to get the source page :
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://dz.q4jobs.com')
html = driver.page_source
soup1 = BeautifulSoup(html_doc)
jobs_list = soup1.find('ul', class_='job_listings')
print(jobs_list)
But this tag keeps showing that I can't get data :
<div class="showing_jobs"></div></form>
<noscript>Your browser does not support JavaScript, or it is disabled. JavaScript must be enabled in order to view listings.</noscript>
<ul class="job_listings loading">
</ul>
Note that I checked the page that selenium opens it when I run the code that javascript is allowed on the browser so I don't get it
I don't see any difficulty performing this with use of basic Selenium WebDriverWait method.
The following code works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = 'https://dz.q4jobs.com/'
driver.get(url)
wait = WebDriverWait(driver, 20)
titles = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".position h3")))
for title in titles:
print(title.text)
The output is:
Freelance Web Designer in Toptal Algeria
Credit Administration Manager in HSBC Algier, Algerien
Freelance React Native Developer in Toptal Algerien
Freelance Blockchain Developer in Toptal Algerien
Community Sales Manager in IWG plc Sétif, Algerien
Freelance Ruby on Rails Developer in Toptal Algerien
Work from home Opportunity in Appen Algier, Algerien
Business English Teacher in CLEVER INSTITUTE DZ Algier, Algerien
Full Stack Engineer, Upland (Remote) – $60,000/year USD in Crossover for Work Algier, Algerien
Développeur web junior in Thinline DZ Algier, Algerien
Business Unit Lead in Philip Morris International Algiers, Algeria
Work from home Opportunity in Appen Algiers, Algeria
Technologues et techniciens/techniciennes en génie mécanique in Choisir le Qubec Algeria
Software Implementation Consultant, CopperTree (Remote) – $60,000/year USD in Crossover for Work Algiers, Algeria
Assistant Store Manager – Banter by Piercing Pagoda – Arbor Place in Signet Jewelers Zaouiet Kounta, Algeria
Pharmacien Assistant DT in LDM Groupe Constantine, Algeria
Freelance eCommerce Developer in Toptal Algeria
Work from home Opportunity in Appen Algiers, Algeria
Développeur full stack .net core in ZS CONSULTING France Algiers Center, Algeria, Algeria
Subject Matter Expert – Software Engineering in Coursera Delhi, Delhi, India
Senior Nodejs Engineer in Time Doctor Brazil
Commercial in Nreservicom Algiers Center, Algeria, Algeria
Freelance Full-Stack Developer in Toptal Algeria
Client Implementation Project Manager in HH Global Mumbai Metropolitan Region
Quality Assurance Analyst in SOLJIT Algiers, Algeria
Commercial Sales Director, EMEA (Remote, EMEA) in Grafana Labs Amsterdam, North Holland, Netherlands
Selfies Data Collection Project – Algeria in TELUS International Algiers, Algeria
Accounting Lead-KFC in Azadea Group Algeria, Algeria
Data Operation Specialist H/F in Data Impact by NielsenIQ Algiers Center, Algeria, Algeria
Remote Project for Arabic Speakers in Appen Algiers Center, Algeria, Algeria
Clinical Operations Hub Regional Country Study Manager in Roche Zéralda, Algeria, Algeria
Freelance Flutter Developer in Toptal Algeria
Sales Manager M/F – Algeria & Tunisia in Michael Page Algiers, Algeria
Mobile iOS Engineer in Yassir (YC W20) Algiers, Algeria
CSC Engineer in Speedcast Algiers, Algeria
Search Evaluator in Alegria in Appen Batna, Algeria
Key Account Specialist/ Est in Sandoz Algiers, Algeria
Customer Service Engineer In vitro in Siemens Healthineers Algiers Center, Algeria, Algeria
Network Account Manager Oncology in Pfizer Algiers, Algeria
Pricing & profitability manager in Yassir (YC W20) Algiers, Algeria
Experienced Frontend Developer in Mindera Algiers Center, Algeria, Algeria
Credit Officer in Aramex Ouled Ahmed Timmi, Adrar, Algeria
District Representative III in Ecolab Algiers, Algeria
Product Manager in Yassir (YC W20) Algiers, Algeria
Freelance Part Time Audio Transcription Project | Arabic Speaker in Appen Bab Ezzouar, Algeria, Algeria
Work From Home Opportunities at Appen in Appen Oran, Algeria
Digital Acquisition specialist in Yassir (YC W20) Algiers, Algeria
Finish Line Macy’s Store Associate – Puerto Rico Premium Outlet, Barceloneta, PR in Finish Line Tkoutt, Batna, Algeria
Senior Software Engineer (Full-Stack) in Think-it Algiers Center, Algeria, Algeria
Freelance UX Designer in Toptal Algeria
Freelance UI Designer in Toptal Algeria
Chef de Produit Marketing Local H/F in Pierre Fabre Group Algiers, Algeria
Creative Director in Yassir (YC W20) Algiers, Algeria
Data, CRM and Lifecycle Marketing Tech Manager in Yassir (YC W20) Algiers, Algeria
Business Developpement professional in Siemens Healthineers Algiers Center, Algeria, Algeria
Freelance Unity Developer in Toptal Algeria
Ingénieur Qualification/ Validation H/F in Anywr Hydra, Algeria, Algeria
Freelance Terraform Developer in Toptal Algeria
Freelance Web Designer in Toptal Algeria
Study Start – Up Lead client – dedicated based in Ukraine in Syneos Health Algiers, Algeria
Freelance Unity Developer in Toptal Algeria
Sales And Marketing Representative in Yangtze Optical Fibre and Cable Joint Stock Limited Company(YOFC) Cezayir, Cezayir
Freelance Part Time Audio Transcription Project | Arabic Speaker in Appen Awlad Addouf, Batna, Algeria
O&M Manager in i engineering Group Algiers, Algeria
Search Evaluator in Alegria in Appen Algiers, Algeria
Executive assistant in Ontex Rouïba, Algeria, Algeria
Insurance Executive in Indépendant Algiers, Algeria
Search Evaluator in Alegria in Appen Awlad Addouf, Batna, Algeria
JUNIOR TRADER – Dubai, UAE in Cobblestone Energy Algiers, Algeria
Freelance Data Engineer in Toptal الجزائر
iOS Developer – Remote in OBSS الجزائر الجزائر الوسطى الجزائر
Freelance Drupal Developer in Toptal Algeria
Freelance Swift Developer in Toptal Algeria
Freelance Front-end Developer in Toptal Algeria
Sales Operations Specialist (Senior) in FENG GROUP Algiers, Algeria
Technical Artist in Nyan Heroes United Kingdom
Senior Account Executive – Enterprise in Recurly England, United Kingdom
Full Stack Engineer in Maze Toronto, Ontario, Canada
Search Ads Evaluator | Work from Home in Appen San Jose, CA
Selfies Collector (Algeria) in TELUS International Algiers, Algeria
Partner Manager, EMEA in Harness Cape Town, Western Cape, South Africa
Sr. Compensation Partner in Zapier Austin, TX
Full-Stack Engineer, Developer Console in Near Inc Delhi, India
Systems Engineer in Solera, Inc. Panama City, Panamá, Panama
Frontend Developer in SARL BJD IT Constantine, Algeria
Temenos – Automation Tester in Skillfinder International Edinburgh, Scotland, United Kingdom
Backend Developer in LumiQualis Algiers Center, Algeria, Algeria
Global Lead DTI Business Partner – Finance in Fresenius Medical Care North America Heidenheim an der Brenz, Baden-Württemberg, Germany
Freelance React Native Developer in Toptal Algeria
Freelance Site Reliability Engineer in Toptal Algeria
Freelance React Developer in Toptal Algeria
Freelance Part Time Audio Transcription Project | Arabic Speaker in Appen Constantine, Algeria
Director of Payroll Products in Omnipresent Tipperary, County Tipperary, Ireland
Freelance Go Developer in Toptal Algeria
Hard Surface Character Artist in Nyan Heroes Orlando, FL
Solutions Engineer, Gaming in Immutable India
DevOps Support Engineer in Infovista Makati, National Capital Region, Philippines
DevOps Engineer in Gigster Raleigh, NC
Quality Rater in Appen United States
Freelance Ruby on Rails Developer in Toptal Algeria