I am trying to scrape Wikipedia Table using Python & Beautiful Soup. When I try to get the table columns attributes with a for loop I get the error:
NameError Traceback (most recent call last)
<ipython-input-18-948408e65d8d> in <module>
1 # Header attributes of the table
2 header=[th.text.rstrip()
----> 3 for th in rows[0].find_all('th')]
4 print(header)
5 print('------------')
NameError: name 'rows' is not defined
How can I fix this?
url="https://en.wikipedia.org/wiki/List_of_municipalities_of_Norway"
Initiating multiple URL requests. If the request is successful, then expected HTTP response status code is 200.
s=requests.Session()
response=s.get(url, timeout=10)
response
res = requests.get(url)
soup = BeautifulSoup(res.content, 'html.parser')
Title of wikipedia page
soup.title.string
Get the right table to scrape
right_table=soup.find('table',{"class":'sortable wikitable'})
Header attributes of the table
header=[th.text.rstrip()
for th in rows[0].find_all('th')]
print(header)
print('------------')
print(len(header))
You might use pandaswhich will be very straight forward in your case:
import pandas as pd
tables = pd.read_html("https://en.wikipedia.org/wiki/List_of_municipalities_of_Norway")
right_table = tables[1]
Output
| | Number[1](ISO 3166-2:NO) | Name | Adm. center | County | Population(2017)[2] | Area(km²)[3] | CountyMap | Arms | Language form[4] | Mayor[5] | Party |
|----:|---------------------------:|:-----------------------------|:---------------------|:---------------------|----------------------:|---------------:|------------:|-------:|:-------------------------|:----------------------------|:--------|
| 0 | 301 | Oslo | Oslo | Oslo | 673469 | 454.03 | nan | nan | Neutral | Marianne Borgen | SV |
| 1 | 1101 | Eigersund | Egersund | Rogaland | 14898 | 431.66 | nan | nan | Bokmål | Leif Erik Egaas | H |
| 2 | 1103 | Stavanger | Stavanger | Rogaland | 141186 | 262.52 | nan | nan | Bokmål | Kari Nessa Nordtun | Ap |
| 3 | 1106 | Haugesund | Haugesund | Rogaland |