I'm scraping a random web page with Scrapy. I created the project, but when I tried to run my spider, it couldn't import scrapy.
This is my spider
import scrapy
import logging
# Spider for truecar.com
class TruecarSpider(scrapy.Spider):
name = "truecar"
def start_requests(self):
urls = ['https://www.truecar.com/used-cars-for-sale/listings/tesla/model-3/']
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
all_listings = response.xpath('//div[@data-test="allVehicleListings"] > ul')
for tesla in all_listings:
aux = tesla.xpath('//div[@class="linkable card card-shadow vehicle-card"]')
make_model = aux.xpath('@aria-label')#aux.css('::attr(aria-label)')
year = make_model.xpath('@aria-label').get()
model_raw = make_model.css('span.vehicle-header-make-model').get()
model = model_raw[model_raw.find('>')+1:-7]. replace ('<! ----3"')
tesla_data = {
'url': 'http://truecar.com' + tesla.css('a::attr (href)').get(),
'model': year + ' ' + model,
'mileage': tesla.css('div[data-test="cardContent"] > div > div. text-truncate: :text').get(),
'price': tesla.css('h4: :text').get(),
}
yield tesla_data
I have it installed through pip install scrapy so I tried through the terminal on VS Code to check if it exists.
This is what I got
I have python 3.10.1 and scrapy 2.7.1
Thanks to @MatBailie's comments, it helped me to see that my python version on the IDE was wrong. I'm using VS Code and if any of you have the same problem you can check it here, on the blue bar below the terminal. If it doesn't appear, make sure you have Python extension and that it is reloaded: