pythonwebpython-requestsscreen-scrapingpython-requests-html
Requests-html not getting all links
I am trying to scrape a website, but it looks I can't acces all links. The website is:
The procedure I am following is first identify each separate product, and then get the link for each product. To my surprise I can identify all the products in the page, but I can only get the link for the first 8, althogh the others should have a link too. My code is:
from requests_html import HTMLSession
s = HTMLSession()
url = "https://www.carrefour.es/supermercado/bebidas/refrescos/colas/cat650010/c?ic_source=portal-y-corporativo&ic_medium=menu-links&ic_content=ns"
r = s.get(url)
products = r.html.find('ul.product-card-list__list li')
for item in products:
print(item.find('a', first=True).attrs["href"])
At some point I get the following error, since I can't find the link of the product, although it exists and the product seems to be loaded:
AttributeError: 'NoneType' object has no attribute 'attrs'
Any hints about where the problem is? Many thanks!!
Solution
Possibly is a js rendering and you are using just downloading the HTML page content. Try use the scrapinghub splash to evaluate, I am from a country where the site is blocked and cannot help much.
- Get the errorbars to bins of a dataset by using bootstrapping
- How to tune PID controller on non-linear model (here: diving of a body)
- django import error - No module named core.management
- Changing tense of text from present/future to past tense
- Discord.py modal form command has no response despite no errors in console
- AttributeError: module 'pyperclip' has no attribute 'waitForPaste'
- Array lists of HTML elements in order by website in selenium / beautiful soup
- Recursive types in Python and difficulties inferring the type of `type(x)(...)`
- `AttributeError` constructing `torchvision.io.VideoReader` in Google Colab
- How to use python selenium to click this element with text bb1
- Base 62 conversion
- How do I get the object if it exists, or None if it does not exist in Django?
- Sampling n= 2000 from a Dask Dataframe of len 18000 generates error Cannot take a larger sample than population when 'replace=False'
- Set same scale in legend matplotlib
- Replicate virtualenv without downloading all the packages again on the same machine
- Tkinter entry widget input are written backwards
- Pydantic v2 vs requests POST JSON - 'not serializable' nightmares
- Unexpected generator behaviour when not assigned to a variable
- How to change the cursor in Pygame to a custom image
- Azure functions in Python - "Duplex option is required" error
- model not showing up in django admin
- How can I see the entire HTTP request that's being sent by my Python application?
- Appending to a list by adding to the previous value in Python
- Layer 'conv2d_11' expected 2 variables, but received 0 variables during loading. Expected: ['conv2d_11/kernel:0', 'conv2d_11/bias:0']
- The Python dbus package is not installed
- How to determine pid of process started via os.system
- Can I store a Parquet file with a dictionary column having mixed types in their values?
- I cannot import monsoon after installing Monsoon
- PyCharm: Why "Python Console" is not accessing ~\.aws\credentials file? How to set it within "Python Console"
- Unexpected Behavior of pd.Grouper with datetime Key and freq Argument