Search code examples
pythonweb-scrapingwebbeautifulsoup

how to remove list brackets from the web scraped data


i'm trying to scrape some data from a website but the final result have the output data in lists, so how can i extract the data without those list brackets.

enter image description here

The Original Code:-

user_input =  'ios-phones'#input('Please Enter Your Favorite Item:- ')
try:
    data_list = []
    for i in range(1,30):

        url = f'https://www.jumia.com.eg/{user_input}/?page={i}#catalog-listing'
        page = requests.get(url).content
        soup = BeautifulSoup(page,'lxml')
        phones = soup.find('div',class_='-paxs row _no-g _4cl-3cm-shs')
        phones_info = phones.find_all('article',class_=True)

        for i in phones_info:
            try:
                title = i.select('.name')[0].text.strip()
                current_price = i.select('.prc')[0].text
                old_price = i.find('div',class_='old')
                rating = i.find('div',class_='stars')
            except:
                pass

            row = {'Phone Title':title,'Current Price':current_price,'Old Price':old_price,'Rating':rating}
            data_list.append(row)
except:
    pass
df = pd.DataFrame(data_list)
df

Solution

  • Main issue here seems to be that you append the bs4 objects for old_price and rating not its texts as you do with the first two title and current_price - So change to:

    for i in phones_info:
        title = i.select_one('.name').get_text(strip=True)
        current_price = i.select_one('.prc').get_text(strip=True)
        old_price = i.select_one('.old').get_text(strip=True) if i.select_one('.old') else None
        rating = i.select_one('.stars').get_text(strip=True) if i.select_one('.stars') else None
    
        row = {'Phone Title':title,'Current Price':current_price,'Old Price':old_price,'Rating':rating}
        data_list.append(row)
    

    Output

    Phone Title Current Price Old Price Rating
    0 Apple Iphone 14 Pro Max – 5G Single SIM – 256/6GB RAM – Deep Purple EGP 54,499.00 EGP 70,000.00 None
    1 Apple IPhone 13 Single SIM With FaceTime - 128GB - Pink EGP 29,999.00 EGP 50,000.00 5 out of 5
    2 Apple IPhone 13 Pro Max Single SIM With FaceTime - 512GB - AlpineGreen EGP 55,499.00 EGP 65,000.00 None
    3 Apple IPhone 12 Mini With FaceTime - 128GB - Blue EGP 25,900.00 None 4.7 out of 5
    4 Apple IPhone 12 With FaceTime - 128GB - Purple EGP 27,900.00 None 4.2 out of 5
    ...
    35 Apple Iphone 13 128G Green EGP 31,900.00 None None
    36 Apple IPhone 13 / 512GB / Pink EGP 40,900.00 None None
    37 Apple IPhone 13 (128GB) Red EGP 31,900.00 None None
    38 Apple IPhone 13 Pro Single SIM With FaceTime - 128GB - Sierra Blue EGP 41,900.00 None None
    39 Apple IPhone 13 128GB Starlight EGP 31,900.00 None None