Search code examples
pythonxpathscrapyscrapy-shell

Xpath not returning the TEXT form the <p> tags in Scrapy Shell

Link: https://www.softwareadvice.com/hr/zenefits-profile

I am trying to scrape the description from the above link. The XPath seems correct but it doesn't return me the value in scrapy shell. (Please see the screenshot below). I tried all methods like get(), getall(), extract(), extract_first(), extractall() but I am getting an empity list. Kindly help me to identify the error. Thanks...

Click to see the image (XPath)

Click to see the image (Scrapy Shell)

Solution

  • If you disable JS you will not find that XPATH working.

    That is how Scrapy loads the HTML, it loads only HTML and does not execute any JS/AJAX

    Try this XPATH 

    response.xpath("/html/body/app-root/main/app-product/div[1]/app-product-detail/div[2]/div/div[1]/div/div/p//text()").getall()