Python: AttributeError: 'bytes' object has no attribute 'find_all'

Question

Just been testing around trying to get web scraping to work but this is bugging me. This is my code.

import requests
from bs4 import BeautifulSoup
from csv import writer

page = requests.get('https://www.ebay.com/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=pc&_sacat=0')

soup = BeautifulSoup(page.text,'html.parser').encode("utf-8")
posts = soup.find_all(class_='s-item__wrapper clearfix')

with open('ebay.csv', 'w') as csv_file:
    csv_writer = writer(csv_file)
    headers = ['Title', 'Price', 'Link']
    csv_writer.writerow(headers)

    for post in posts:
        price = post.find(class_='s-item__price').get_text().replace('\n', '')
        title = post.find(class_='s-item__title').get_text().replace('\n', '')
        link = post.find('a')['href']
        csv_writer.writerow([title, price, link])

I keep getting this error

Traceback (most recent call last):
  File "path/WebScraping.py", line 8, in <module>
    posts = soup.find_all(class_='s-item__wrapper clearfix')
AttributeError: 'bytes' object has no attribute 'find_all'

Tried to find other solutions but can't find any that work for me. The code works but only for a third of the page.

Answer 1

You are trying to encode the soup object which produces bytes object representation of that object and there is no method named find_all in bytes object.

Replace:

soup = BeautifulSoup(page.text,'html.parser').encode("utf-8")

With:

soup = BeautifulSoup(page.text,'html.parser')