Search code examples
pythonseleniumcsvweb-scrapingexport-to-csv

Formatting Issue in csv file while trying to scrape the following website


I am trying to scrape a website to get the title and prices but once the data is extracted and saved on the csv file the prices column formatting get disturbed and is not properly displayed in the column e.g $8,900 become $8 in one column and 900 is shifted to next column.

from selenium import webdriver
import time

max_pages = 1
driver = webdriver.Chrome()
with open('autotrader.csv', 'w') as f:
    f.write("Title,Price \n")

for i in range(1, max_pages + 1):
    url =  "https://www.autotrader.co.uk/car-search?advertClassification=standard&postcode=WC2N%205DU&onesearchad=Used&onesearchad=Nearly%20New&onesearchad=New&advertising-location=at_cars&is-quick-search=TRUE&include-delivery-option=on&page=" + str(max_pages)

driver.get(url)
title = driver.find_elements_by_xpath('//h3[@class="product-card-details__title"]')
price =driver.find_elements_by_xpath('//div[@class="product-card-pricing__price"]')
page_items = len(title)
with open('autotrader.csv', 'a') as f:
    for i in range(page_items):
        f.write(title[i].text + "," + price[i].text + "\n")
driver.close()

Solution

  • Use csv.writer and it will properly quote fields with delimiter characters in them:

    import csv
    
    # ... code to fetch titles and prices ...
    
    with open('autotrader.csv', 'w', newline='') as f:
        w = csv.writer(f)
        w.writerow(['Title','Price'])
        for t,p in zip(title,price):
            w.writerow([t.text,p.text])