pythonweb-scrapingbeautifulsoupdropdownhtml-select
Scraping data from a site where the URL doesn't change while changing options in a drop-down list
I'm using BeautifulSoup to scrape the table Antwerp Weather History for 1 April 2017 in this webpage. But I don't only need this date, I need all days in April 2017, which are in a drop-down list:
In the inspector, it's a select tag with options like these:
I could get the values of them with the next code:
prefix = 'https://www.timeanddate.com'
weather_request = requests.get(prefix + '/weather/belgium/antwerp/historic?month=4&year=2017',
'html.parser')
weather = BeautifulSoup(weather_request.content)
for option in weather.select('select > option'):
append_to_mylist(option.get('value'), option.text)
Would you help me, how to scrape the tables beyond these values, as the URL doesn't change while changing the option from the drop-down list?
I've found some other similar questions but weren't about BeautifulSoup
Solution
The data is loaded via Ajax from other URL. The data returned is not Json, but raw Javascript, so some preprocessing is needed to parse it correctly.
For example:
import re
import json
import requests
import pandas as pd
from bs4 import BeautifulSoup
for day in range(1, 31):
print('Getting info for day {}..'.format(day))
url = 'https://www.timeanddate.com/scripts/cityajax.php?n=belgium/antwerp&mode=historic&hd=201704{:02d}&month=4&year=2017&json=1'.format(day)
data = requests.get(url).text
data = json.loads(re.sub(r'(c|h|s):', r'"\1":', data))
# uncomment this to print raw data:
# print(json.dumps(data, indent=4))
# construct the table from json:
table = '<table>'
for row in data:
table += '<tr>'
for cell in row['c']:
table += '<td>' + BeautifulSoup(cell['h'], 'html.parser').get_text(strip=True, separator=' ') + '</td>'
table += '</tr>'
table += '</table>'
# now in `table` is HTML table, you can parse it with BeautifulSoup, or pass it to Pandas:
df = pd.read_html(table)[0]
print(df)
print('-' * 120)
Prints:
Getting info for day 1..
0 1 2 3 4 5 6 7 8
0 12:20 am Sat, Apr 1 NaN 50 °F Clear. 2 mph ↑ 94% 29.92 "Hg 2 mi
1 12:50 am NaN 46 °F Fog. 2 mph ↑ 100% 29.92 "Hg 2 mi
2 1:20 am NaN 48 °F Light fog. 3 mph ↑ 87% 29.89 "Hg 0 mi
3 1:50 am NaN 48 °F Clear. 3 mph ↑ 94% 29.89 "Hg 1 mi
4 2:20 am NaN 46 °F Fog. 5 mph ↑ 100% 29.89 "Hg 1 mi
5 3:20 am NaN 46 °F Clear. 3 mph ↑ 93% 29.89 "Hg 1 mi
6 3:50 am NaN 46 °F Fog. 6 mph ↑ 93% 29.86 "Hg 1 mi
7 4:20 am NaN 46 °F Fog. 3 mph ↑ 100% 29.86 "Hg 1 mi
8 4:50 am NaN 46 °F Fog. 3 mph ↑ 100% 29.86 "Hg 1 mi
9 5:20 am NaN 46 °F Fog. 2 mph ↑ 93% 29.86 "Hg 2 mi
10 5:50 am NaN 48 °F Clear. 3 mph ↑ 87% 29.86 "Hg 4 mi
11 6:20 am NaN 48 °F Clear. 5 mph ↑ 87% 29.83 "Hg 4 mi
12 6:50 am NaN 48 °F Clear. 5 mph ↑ 94% 29.86 "Hg 4 mi
13 7:20 am NaN 50 °F Sprinkles. Clear. 6 mph ↑ 94% 29.86 "Hg 4 mi
14 7:50 am NaN 52 °F Sprinkles. Broken clouds. 9 mph ↑ 88% 29.86 "Hg 3 mi
15 8:20 am NaN 52 °F Light rain. Partly sunny. 8 mph ↑ 88% 29.86 "Hg 5 mi
16 8:50 am NaN 52 °F Light rain. Passing clouds. 6 mph ↑ 94% 29.86 "Hg 5 mi
17 9:20 am NaN 52 °F Drizzle. Partly sunny. 5 mph ↑ 94% 29.86 "Hg 5 mi
18 9:50 am NaN 52 °F Broken clouds. 5 mph ↑ 94% 29.86 "Hg 5 mi
19 10:20 am NaN 52 °F Broken clouds. 6 mph ↑ 94% 29.89 "Hg NaN
20 10:50 am NaN 52 °F Sprinkles. Broken clouds. 8 mph ↑ 94% 29.89 "Hg 5 mi
21 11:20 am NaN 52 °F Partly sunny. 5 mph ↑ 94% 29.89 "Hg NaN
22 11:50 am NaN 54 °F Scattered clouds. 2 mph ↑ 88% 29.89 "Hg NaN
23 12:20 pm NaN 55 °F Scattered clouds. 5 mph ↑ 82% 29.89 "Hg NaN
24 12:50 pm NaN 55 °F Scattered clouds. 3 mph ↑ 77% 29.89 "Hg NaN
25 1:20 pm NaN 57 °F Passing clouds. 5 mph ↑ 72% 29.89 "Hg NaN
26 1:50 pm NaN 57 °F Passing clouds. 3 mph ↑ 67% 29.89 "Hg NaN
27 2:20 pm NaN 57 °F Passing clouds. 7 mph ↑ 72% 29.89 "Hg NaN
28 2:50 pm NaN 57 °F Scattered clouds. 3 mph ↑ 72% 29.89 "Hg NaN
29 3:20 pm NaN 55 °F Sprinkles. Broken clouds. 9 mph ↑ 77% 29.89 "Hg 4 mi
30 3:50 pm NaN 55 °F Sprinkles. Broken clouds. 3 mph ↑ 77% 29.86 "Hg 5 mi
31 4:20 pm NaN 55 °F Sprinkles. Broken clouds. 2 mph ↑ 82% 29.89 "Hg NaN
32 4:50 pm NaN 57 °F Scattered clouds. 2 mph ↑ 77% 29.86 "Hg NaN
33 5:20 pm NaN 57 °F Scattered clouds. 7 mph ↑ 72% 29.89 "Hg NaN
34 5:50 pm NaN 55 °F Scattered clouds. 6 mph ↑ 88% 29.89 "Hg NaN
35 6:20 pm NaN 55 °F Passing clouds. 6 mph ↑ 82% 29.89 "Hg NaN
36 6:50 pm NaN 55 °F Passing clouds. 3 mph ↑ 82% 29.89 "Hg NaN
37 7:20 pm NaN 54 °F Passing clouds. 5 mph ↑ 94% 29.89 "Hg NaN
38 7:50 pm NaN 54 °F Passing clouds. 5 mph ↑ 88% 29.89 "Hg NaN
39 8:20 pm NaN 54 °F Passing clouds. 7 mph ↑ 88% 29.92 "Hg NaN
40 8:50 pm NaN 54 °F Clear. 7 mph ↑ 88% 29.92 "Hg 10 mi
41 9:20 pm NaN 54 °F Clear. 2 mph ↑ 88% 29.92 "Hg 10 mi
42 9:50 pm NaN 52 °F Clear. 5 mph ↑ 94% 29.92 "Hg 10 mi
43 10:20 pm NaN 48 °F Clear. 2 mph ↑ 100% 29.95 "Hg 10 mi
44 10:50 pm NaN 52 °F Clear. 3 mph ↑ 88% 29.95 "Hg 4 mi
45 11:20 pm NaN 46 °F Fog. 2 mph ↑ 93% 29.95 "Hg 1 mi
46 11:50 pm NaN 46 °F Clear. 3 mph ↑ 93% 29.95 "Hg 0 mi
------------------------------------------------------------------------------------------------------------------------
Getting info for day 2..
0 1 2 3 4 5 6 7 8
0 12:20 am Sun, Apr 2 NaN 45 °F Fog. 2 mph ↑ 100% 29.95 "Hg 0 mi
1 12:50 am NaN 45 °F Fog. 2 mph ↑ 93% 29.98 "Hg 1 mi
2 1:20 am NaN 45 °F Fog. 2 mph ↑ 100% 29.95 "Hg 0 mi
3 1:50 am NaN 45 °F Clear. 3 mph ↑ 87% 29.98 "Hg 4 mi
4 2:20 am NaN 48 °F Clear. 6 mph ↑ 87% 29.98 "Hg 10 mi
5 2:50 am NaN 48 °F Clear. 2 mph ↑ 87% 29.98 "Hg 10 mi
6 3:20 am NaN 48 °F Clear. 5 mph ↑ 87% 29.98 "Hg 10 mi
7 3:50 am NaN 48 °F Clear. 2 mph ↑ 87% 29.98 "Hg 6 mi
8 4:50 am NaN 46 °F Clear. 2 mph ↑ 87% 30.01 "Hg 10 mi
9 5:20 am NaN 46 °F Passing clouds. 3 mph ↑ 87% 30.01 "Hg NaN
10 5:50 am NaN 46 °F Clear. 2 mph ↑ 87% 30.01 "Hg 10 mi
11 6:20 am NaN 46 °F Clear. 1 mph ↑ 87% 30.04 "Hg 4 mi
12 6:50 am NaN 45 °F Light fog. 2 mph ↑ 93% 30.04 "Hg 5 mi
... and so on.
- Python method chaining in functional programming style
- flask-jwt-extended: Fake Authorization Header during testing (pytest)
- For loop through the list unless empty?
- Polars make all groups the same size
- Is there a way to specify a default base-template for all templates in django?
- How to tackle time limit exceeded error in leetcode
- Is pd.get_dummies() updated in newer versions of Pandas making it default to Booleans (True/False) instead of (0/1)?
- What's the function like sum() but for multiplication? product()?
- How to type hint a dynamically-created dataclass
- Issue with pulling the data with EIA API with Python
- 403 Forbidden Error when scraping a site, user-agents already used and updated. Any ideas?
- Fullstack web-hosting services
- How to handle an AnalysisException on Spark SQL?
- Python requests is slow and takes very long to complete HTTP or HTTPS request
- Is there a way to modify an element in a Numpy array based on the value of other elements?
- Tkinter grid manager height/width nonconsistent
- Sql Alchemy Insert Statement failing to insert, but no error
- How can I create a Polars struct while eval-ing a list?
- Excel using win32com and python
- django: on pypy, psyco, unladen swallow or cpython, which one is the fastest?
- How convert a list into multiple columns and a dataframe?
- Name not defined in type annotation
- Static type checkers and Language Servers not recognizing attributes of objects that are subclasses
- How do I get multiple OID values in PySNMP?
- how to log a file in Django
- Is there a simple and efficient way to evaluate Elementary Symmetric Polynomials in Python?
- Iterating over two lists one after another
- Python -i flag for production
- What is PyCompilerFlags in Python C API?
- How to make Python check whether a variable is a number or letter