I'm scraping a web (using Python requests and requests-html modules) and I need to go through all the pages of a list of items.
In "human user" world, I click either in the "2" for second page, or in "->" to go from actual to next page.
When I inspect the elements I just said, they are a <div> tag such as:
<div class="pagination__Page..."> 2 </div> or
<div class="pagination__Page..."> -> </div>
Both with an event linked to each, so that when I click on it, moves to the following page.
I've tried to do the for loop pagination suggested by requests-HTML documentation but it doesn't work in this case, because there are no links associated to the r.html object, nor to each page of the list.
When I click, in the website, on those "divs" the url does not change at all.
Inspecting the event (for the 2 case) it calls a JS function, such as:
function() {
return a({
pageNum: e
})
}
Inspecting the event function (for the -> case) it calls a JS, such as:
function() {
return a({
direction: "right"
})
}
I want to get the same result as when clicked, but I don't know how.
You'll have to use the dev tools to get the precise query parameters (specifically the rqid), but this should get you going. It'll return the full list, without need to go page by page:
import requests
from pandas.io.json import json_normalize
url = 'https://www.flightstats.com/v2/api-next/flight-tracker/arr/ORY/2019/4/29/6'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
query = {
'carrierCode': '',
'numHours': '6',
'rqid': '7tl8o43bkps'}
jsonData = requests.get(url, headers=headers, params=query).json()
df = json_normalize(jsonData['data']['flights'])
Output:
print (df)
airport.city ... url
0 Cayenne ... /flight-tracker/TX/571?year=2019&month=4&date=...
1 Saint Denis de la Reunion ... /flight-tracker/AF/671?year=2019&month=4&date=...
2 Pointe-a-Pitre ... /flight-tracker/SS/3541?year=2019&month=4&date...
3 Pointe-a-Pitre ... /flight-tracker/TX/541?year=2019&month=4&date=...
4 Moscow ... /flight-tracker/S7/4021?year=2019&month=4&date...
5 Moscow ... /flight-tracker/ZI/516?year=2019&month=4&date=...
6 Cayenne ... /flight-tracker/AF/853?year=2019&month=4&date=...
7 Cayenne ... /flight-tracker/KL/2245?year=2019&month=4&date...
8 Toulouse ... /flight-tracker/AF/6101?year=2019&month=4&date...
9 Pointe-a-Pitre ... /flight-tracker/KL/2261?year=2019&month=4&date...
10 Toulouse ... /flight-tracker/HOP/5101?year=2019&month=4&dat...
11 Pointe-a-Pitre ... /flight-tracker/AF/793?year=2019&month=4&date=...
12 Beirut ... /flight-tracker/SS/6628?year=2019&month=4&date...
13 Beirut ... /flight-tracker/ZI/628?year=2019&month=4&date=...
14 Montpellier ... /flight-tracker/AF/7541?year=2019&month=4&date...
15 Geneva ... /flight-tracker/U2/1399?year=2019&month=4&date...
16 Montpellier ... /flight-tracker/HOP/5541?year=2019&month=4&dat...
17 Ajaccio ... /flight-tracker/AF/4442?year=2019&month=4&date...
18 Bastia ... /flight-tracker/HOP/7780?year=2019&month=4&dat...
19 Ajaccio ... /flight-tracker/HOP/7770?year=2019&month=4&dat...
20 Ajaccio ... /flight-tracker/XK/770?year=2019&month=4&date=...
21 Bastia ... /flight-tracker/XK/780?year=2019&month=4&date=...
22 Bastia ... /flight-tracker/AF/4458?year=2019&month=4&date...
23 Marseille ... /flight-tracker/HOP/5001?year=2019&month=4&dat...
24 Marseille ... /flight-tracker/AF/6001?year=2019&month=4&date...
25 Clermont-Ferrand ... /flight-tracker/AF/7433?year=2019&month=4&date...
26 Clermont-Ferrand ... /flight-tracker/HOP/5433?year=2019&month=4&dat...
27 Bordeaux ... /flight-tracker/HOP/5253?year=2019&month=4&dat...
28 Bordeaux ... /flight-tracker/AF/6253?year=2019&month=4&date...
29 Nice ... /flight-tracker/HOP/5203?year=2019&month=4&dat...
.. ... ... ...
192 Marseille ... /flight-tracker/HOP/5009?year=2019&month=4&dat...
193 Sevilla ... /flight-tracker/TO/3201?year=2019&month=4&date...
194 Bordeaux ... /flight-tracker/AF/6277?year=2019&month=4&date...
195 Toulouse ... /flight-tracker/U2/4026?year=2019&month=4&date...
196 Toulouse ... /flight-tracker/HOP/5117?year=2019&month=4&dat...
197 Toulouse ... /flight-tracker/AF/6117?year=2019&month=4&date...
198 Rome ... /flight-tracker/IB/5193?year=2019&month=4&date...
199 Rome ... /flight-tracker/VY/6251?year=2019&month=4&date...
200 Bordeaux ... /flight-tracker/HOP/5277?year=2019&month=4&dat...
201 Faro ... /flight-tracker/U2/4278?year=2019&month=4&date...
202 Campinas ... /flight-tracker/AD/8900?year=2019&month=4&date...
203 Casablanca ... /flight-tracker/AT/760?year=2019&month=4&date=...
204 Campinas ... /flight-tracker/ZI/36?year=2019&month=4&date=2...
205 Rome ... /flight-tracker/U2/4242?year=2019&month=4&date...
206 Ajaccio ... /flight-tracker/XK/772?year=2019&month=4&date=...
207 Ajaccio ... /flight-tracker/AF/4445?year=2019&month=4&date...
208 Ajaccio ... /flight-tracker/HOP/7772?year=2019&month=4&dat...
209 Madrid ... /flight-tracker/AV/6049?year=2019&month=4&date...
210 Madrid ... /flight-tracker/AA/8758?year=2019&month=4&date...
211 Madrid ... /flight-tracker/IB/3436?year=2019&month=4&date...
212 Setif ... /flight-tracker/AH/1108?year=2019&month=4&date...
213 Berlin ... /flight-tracker/ZI/608?year=2019&month=4&date=...
214 Berlin ... /flight-tracker/SS/6608?year=2019&month=4&date...
215 Toulon ... /flight-tracker/AF/7513?year=2019&month=4&date...
216 Toulon ... /flight-tracker/HOP/5513?year=2019&month=4&dat...
217 Perpignan ... /flight-tracker/AF/7465?year=2019&month=4&date...
218 Perpignan ... /flight-tracker/HOP/5465?year=2019&month=4&dat...
219 Rodez ... /flight-tracker/BE/7682?year=2019&month=4&date...
220 Nantes ... /flight-tracker/AF/7383?year=2019&month=4&date...
221 Nantes ... /flight-tracker/HOP/5383?year=2019&month=4&dat...
[222 rows x 13 columns]