So i wanted to get all "b" tags with "price" class form this website, but soup.findAll return empty list with 0 elements. But when I try the same thing, but with local html file, everything just works fine.
I've searched all over the internet, but nothing seems to help.
The code itself:
from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
url = 'https://warframe.market/items/nami_skyla_prime_blueprint'
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage, 'html.parser')
tags = soup.findAll('b')
print(tags)
The data you see on the page is loaded from external URL via JavaScript (so beautifulsoup doesn't see it). To load the data into pandas dataframe you can use this example:
import requests
import pandas as pd
api_url = (
"https://api.warframe.market/v1/items/nami_skyla_prime_blueprint/orders"
)
df = pd.DataFrame(requests.get(api_url).json()["payload"]["orders"])
df = pd.concat([df, df.pop("user").apply(pd.Series)], axis=1)
print(df.to_markdown(index=False))
Prints:
| platform | region | order_type | creation_date | platinum | quantity | last_update | visible | id | reputation | region | avatar | last_seen | ingame_name | id | status |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| pc | en | sell | 2018-03-24T13:19:44.000+00:00 | 5 | 1 | 2018-03-24T13:19:44.000+00:00 | True | 5ab6507049ef000fcf1920a9 | 0 | en | 2022-07-30T18:00:20.943+00:00 | lastcrist | 62e56f91330ca40377b19895 | offline | |
| pc | en | sell | 2019-01-10T18:21:06.000+00:00 | 14 | 1 | 2022-07-26T18:45:05.000+00:00 | True | 5c378d12af045700a4291477 | 3 | en | 2022-07-27T11:04:01.133+00:00 | kekxqt | 5a0628ddc611360011c7a66e | offline | |
| pc | en | sell | 2019-12-09T07:11:43.000+00:00 | 35 | 1 | 2021-12-02T04:32:53.000+00:00 | True | 5dedf3afc72155005cf4028c | 968 | en | user/avatar/55f4e189b66f831701dbff77.png?0fe7b910face5429601e7085d289f869 | 2022-07-30T16:18:02.366+00:00 | Sahysa | 55f4e189b66f831701dbff77 | offline |
| pc | en | sell | 2020-02-04T03:29:02.000+00:00 | 13 | 1 | 2022-07-29T23:22:45.000+00:00 | True | 5e38e4fe7b0275005e6d29a9 | 4 | en | 2022-07-30T06:40:38.150+00:00 | YellowSnowBoi | 5b3c268c0c3535012e9b6ee3 | offline | |
| pc | en | sell | 2020-03-29T20:14:39.000+00:00 | 15 | 1 | 2022-07-24T18:14:30.000+00:00 | True | 5e8101aff3441505a6b292c6 | 19 | en | user/avatar/5b2931d4eb069f04986adb6d.png?822ece14d8951796338ef3739d5c6b2c | 2022-07-29T20:55:01.617+00:00 | xblq | 5b2931d4eb069f04986adb6d | offline |
| pc | ko | sell | 2020-11-07T09:15:58.000+00:00 | 9 | 1 | 2020-11-07T09:15:58.000+00:00 | True | 5fa665ced5b8ca0262d04cf0 | 0 | ko | 2022-07-30T17:33:05.154+00:00 | blazespirit | 5a4380778c474d007384823b | ingame | |
| pc | en | sell | 2021-02-07T12:42:27.000+00:00 | 15 | 1 | 2021-12-26T14:35:01.000+00:00 | True | 601fe033ec5e7e01d47859a1 | 3 | en | user/avatar/600fcf3dc59d3703156a15e3.png?5f9404dcf7c36b3200d2eaab5ed9dba9 | 2022-07-30T10:26:53.816+00:00 | pavelfizek | 600fcf3dc59d3703156a15e3 | offline |
| pc | en | buy | 2021-02-12T14:34:49.000+00:00 | 10 | 6 | 2022-07-20T21:11:53.000+00:00 | True | 60269209039a2702632da531 | 412 | en | user/avatar/598d8bf2d3ffb648a838abba.png?dfe50b7091f0e8c44b63976a700eebd9 | 2022-07-30T10:02:13.214+00:00 | Xnidior | 598d8bf2d3ffb648a838abba | offline |
| pc | en | sell | 2021-02-13T03:24:21.000+00:00 | 10 | 1 | 2022-07-23T06:48:55.000+00:00 | True | 602746651c9894026f386fb0 | 2 | en | 2022-07-28T02:31:25.058+00:00 | Reaper575 | 5cd904444532a609fef941db | offline | |
| pc | en | sell | 2021-03-13T18:03:01.000+00:00 | 15 | 1 | 2022-07-22T14:35:07.000+00:00 | True | 604cfe55171b1903005aa375 | 23 | en | user/avatar/5c1cc13e6ad4a90015110a59.png?3eca6f848f62560068b7c0f17f141d6d | 2022-07-27T11:36:30.529+00:00 | -CG-Luthix | 5c1cc13e6ad4a90015110a59 | offline |
| pc | ru | buy | 2021-04-03T05:56:22.000+00:00 | 5 | 1 | 2022-04-07T10:38:12.000+00:00 | True | 606803861a02f00165d2965c | 2 | ru | user/avatar/624ebb41596f2f00881ce522.png?c1556e2a317fe8dfbd7e86ba051e371a | 2022-07-27T14:19:19.788+00:00 | 1FioLet1 | 624ebb41596f2f00881ce522 | offline |
| pc | en | sell | 2021-06-11T13:42:09.000+00:00 | 20 | 1 | 2021-06-11T13:42:09.000+00:00 | True | 60c368311970de00c952915f | 0 | en | 2022-07-30T02:14:54.187+00:00 | NEXUREX21 | 6039915229f04b0128def8fa | offline |
...