I am creating a web-scraping API that will scrape my university's dining hall website and pull the food items that are currently offered depending on the time of day. Currently, the web scraper is not working only on the dining hall page.
Other dynamic websites, where the content does not load if you disable JavaScript, works with the current code (I used a frontend application I've been working on with React).
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
url = 'https://music-bridge.lucagiannotti.com/'
driver = webdriver.Chrome(service=ChromeService(
ChromeDriverManager().install()))
driver.get(url)
elements = driver.find_elements(By.CLASS_NAME, 'container')
for title in elements:
print(title.text)
This code DOES yield all elements under the class container
.
However, if I scrape the dining hall page, it prints nothing, regardless of what class I put in.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
url = 'https://dineoncampus.com/tamu/whats-on-the-menu'
driver = webdriver.Chrome(service=ChromeService(
ChromeDriverManager().install()))
driver.get(url)
print(driver.page_source)
# select elements by class name
elements = driver.find_elements(By.CLASS_NAME, 'table')
for title in elements:
print(title.text)
I figured it might be an issue with the page loading, but after testing this code, that was not the problem either. The full page loads, and all content is visible, but the table
class is never found.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://dineoncampus.com/tamu/whats-on-the-menu'
driver = webdriver.Chrome(service=ChromeService(
ChromeDriverManager().install()))
driver.get(url)
wait = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "table"))
)
# print(driver.page_source)
# select elements by class name
elements = driver.find_elements(By.CLASS_NAME, 'table')
for title in elements:
print(title.text)
I'm sure it's something to do with how the page loads the JavaScript, but I do not know exactly what the problem is, nor have I found anyone solutions with a Google search.
The issue is you're waiting for the ID of table to appear, which isn't going to happen. You need the CLASS_NAME of table, or something like menu-items.
Switching to EC.presence_of_element_located((By.CLASS_NAME, "menu-items"))
:
Grill
Item
Portion
Calories
Eggs Made to Order
2 cage free eggs prepared to order
Nutritional Info
1 each
60
Hash Browns
Nutritional Info
1/2 cup
180
Pork Sausage Link
Spiced breakfast sausage
Nutritional Info
2 each
260
Vegetable Omelet
Nutritional Info
1 each
140
Cheese Omelet
Nutritional Info
5 oz portion
...
Oats 'n Honey Granola
Nutritional Info
1 tbsp
20