me and my friends make a Fifa draft every year and I was looking for a quick way to update player´s info.
What I want is to scrape some information from players in different pages. The pages are:
https://sofifa.com/players?offset=0
https://sofifa.com/players?offset=51
https://sofifa.com/players?offset=101
This is my code:
import requests
from bs4 import BeautifulSoup
import re
pages = []
collection = ['0', '51', '101']
for i in collection:
url = 'https://sofifa.com/players?offset=' + str(i)
pages.append(url)
for item in pages:
page = requests.get(item)
soup = BeautifulSoup(page.text, 'html.parser')
table = soup.find('table', {'class': 'table-hover'})
tbody = table.find('tbody')
info = tbody.find_all('tr')
records = []
for infos in info:
nome = infos.find("a", href=re.compile("/player/")).string
numero = infos.find('img').get('id')
idade = infos.find('div' , {'class': 'col-ae'}).string
ponto = infos.find('div' , {'class': 'col-oa'}).string
potencial = infos.find('div' , {'class': 'col-pt'}).string
records.append((nome, numero, idade, ponto, potencial))
import pandas as pd
df = pd.DataFrame(records, columns=['nome', 'numero', 'idade', 'ponto', 'potencial'])
df.to_excel('jogadores.xls', index=False, encoding='utf-8')
The thing is that I only get the results from page "101", the first tow pages are not analysed. I've searched everywhere here but with no luck.
What I'm confused is that if I print(soup) it will show me the HTML from the three pages!
How can I solve this?
This is and exemple of the HTML code:
<table class="table table-hover persist-area">
<thead>...</thead>
<tbody> <tr>
<td> <figure class="avatar">
<img alt="" data-src="https://cdn.sofifa.org/players/4/19/158023.png" data-srcset="https://cdn.sofifa.org/players/4/19/[email protected] 2x, https://cdn.sofifa.org/players/4/19/[email protected] 3x" src="https://cdn.sofifa.org/players/4/19/158023.png" width="48" height="48" id="158023" class="player-check loaded" srcset="https://cdn.sofifa.org/players/4/19/[email protected] 2x, https://cdn.sofifa.org/players/4/19/[email protected] 3x" data-was-processed="true"></figure></td>
<td>
<div class="col-name text-ellipsis rtl"><a href="/players?na=52" rel="nofollow" title="Argentina"><img alt="" src="https://cdn.sofifa.org/flags/52.png" data-src="https://cdn.sofifa.org/flags/52.png" data-srcset="https://cdn.sofifa.org/flags/[email protected] 2x, https://cdn.sofifa.org/flags/[email protected] 3x" class="flag loaded" style="width:23px;height:17px" srcset="https://cdn.sofifa.org/flags/[email protected] 2x, https://cdn.sofifa.org/flags/[email protected] 3x" data-was-processed="true"></a> <a href="/player/158023" title="Lionel Messi">L. Messi</a> <div class="text-ellipsis rtl"><a rel="nofollow" href="/players?pn=21"><span class="pos pos21">CF</span></a> <a rel="nofollow" href="/players?pn=23"><span class="pos pos23">RW</span></a> <a rel="nofollow" href="/players?pn=25"><span class="pos pos25">ST</span></a></div>
</div>
</td> <td class="col text-center" data-col="ae">
<div class="col-digit col-ae">31</div>
</td> <td class="col text-center" data-col="oa">
<div class="col-digit col-oa"><span class="label p94">94</span></div>
</td> <td class="col text-center" data-col="pt">
<div class="col-digit col-pt"><span class="label p94">94</span></div>
</td> <td>
<div class="col-name text-ellipsis rtl"> <figure class="avatar avatar-sm transparent">
<img alt="" class="team loaded" data-src="https://cdn.sofifa.org/teams/2/19/light/241.png" data-srcset="https://cdn.sofifa.org/teams/2/19/light/[email protected] 2x, https://cdn.sofifa.org/teams/2/19/light/[email protected] 3x" src="https://cdn.sofifa.org/teams/2/19/light/241.png" width="24" height="24" srcset="https://cdn.sofifa.org/teams/2/19/light/[email protected] 2x, https://cdn.sofifa.org/teams/2/19/light/[email protected] 3x" data-was-processed="true">
</figure>
<a href="/team/241">FC Barcelona</a> <div class="subtitle text-ellipsis rtl">2004 ~ 2021</div>
</div>
</td><th class="gap"></th> <td class="col text-center" data-col="vl">
<div class="col-digit col-vl">€110.5M</div>
</td> <td class="col text-center" data-col="wg">
<div class="col-digit col-wg">€565K</div>
</td><th class="gap"></th> <td class="col text-center" data-col="tt">
<div class="col-digit col-tt">2195</div>
</td> <td class="gap"></td>
<td> <div class="col-comments text-right text-ellipsis rtl">0.8K / 32.9K </div></td></tr> <tr>
<td> <figure class="avatar">
<img alt="" data-src="https://cdn.sofifa.org/players/4/19/20801.png" data-srcset="https://cdn.sofifa.org/players/4/19/[email protected] 2x, https://cdn.sofifa.org/players/4/19/[email protected] 3x" src="https://cdn.sofifa.org/players/4/19/20801.png" width="48" height="48" id="20801" class="player-check loaded" srcset="https://cdn.sofifa.org/players/4/19/[email protected] 2x, https://cdn.sofifa.org/players/4/19/[email protected] 3x" data-was-processed="true"></figure></td>
<td>
<div class="col-name text-ellipsis rtl"><a href="/players?na=38" rel="nofollow" title="Portugal"><img alt="" src="https://cdn.sofifa.org/flags/38.png" data-src="https://cdn.sofifa.org/flags/38.png" data-srcset="https://cdn.sofifa.org/flags/[email protected] 2x, https://cdn.sofifa.org/flags/[email protected] 3x" class="flag loaded" style="width:23px;height:17px" srcset="https://cdn.sofifa.org/flags/[email protected] 2x, https://cdn.sofifa.org/flags/[email protected] 3x" data-was-processed="true"></a> <a href="/player/20801" title="C. Ronaldo dos Santos Aveiro">Cristiano Ronaldo</a> <div class="text-ellipsis rtl"><a rel="nofollow" href="/players?pn=25"><span class="pos pos25">ST</span></a> <a rel="nofollow" href="/players?pn=27"><span class="pos pos27">LW</span></a></div>
</div>
</td> <td class="col text-center" data-col="ae">
<div class="col-digit col-ae">33</div>
</td> <td class="col text-center" data-col="oa">
<div class="col-digit col-oa"><span class="label p94">94</span></div>
</td> <td class="col text-center" data-col="pt">
<div class="col-digit col-pt"><span class="label p94">94</span></div>
</td> <td>
<div class="col-name text-ellipsis rtl"> <figure class="avatar avatar-sm transparent">
<img alt="" class="team loaded" data-src="https://cdn.sofifa.org/teams/2/19/light/45.png" data-srcset="https://cdn.sofifa.org/teams/2/19/light/[email protected] 2x, https://cdn.sofifa.org/teams/2/19/light/[email protected] 3x" src="https://cdn.sofifa.org/teams/2/19/light/45.png" width="24" height="24" srcset="https://cdn.sofifa.org/teams/2/19/light/[email protected] 2x, https://cdn.sofifa.org/teams/2/19/light/[email protected] 3x" data-was-processed="true">
</figure>
<a href="/team/45">Juventus</a> <div class="subtitle text-ellipsis rtl">2018 ~ 2022</div>
</div>
</td><th class="gap"></th> <td class="col text-center" data-col="vl">
<div class="col-digit col-vl">€77M</div>
</td> <td class="col text-center" data-col="wg">
<div class="col-digit col-wg">€405K</div>
</td><th class="gap"></th> <td class="col text-center" data-col="tt">
<div class="col-digit col-tt">2228</div>
</td> <td class="gap"></td>
<td> <div class="col-comments text-right text-ellipsis rtl">0.8K / 40.1K </div></td></tr></tbody>
</table>
It's because of the indentation which G_M has already pointed out. Try this way instead:
import requests
from bs4 import BeautifulSoup
import re
pages = []
records = []
collection = ['0', '51', '101']
for i in collection:
url = 'https://sofifa.com/players?offset=' + str(i)
pages.append(url)
for item in pages:
page = requests.get(item)
soup = BeautifulSoup(page.text, 'html.parser')
table = soup.find('table', {'class': 'table-hover'})
tbody = table.find('tbody')
info = tbody.find_all('tr')
for infos in info:
nome = infos.find("a", href=re.compile("/player/")).string
numero = infos.find('img').get('id')
idade = infos.find('div' , {'class': 'col-ae'}).string
ponto = infos.find('div' , {'class': 'col-oa'}).string
potencial = infos.find('div' , {'class': 'col-pt'}).string
records.append((nome, numero, idade, ponto, potencial))
print(nome,numero,idade,ponto,potencial)
import pandas as pd
df = pd.DataFrame(records, columns=['nome', 'numero', 'idade', 'ponto', 'potencial'])
df.to_excel('jogadores.xls', index=False, encoding='utf-8')