Search code examples
pythonweb-scrapingscrapy

Scrapy Body Text Only

I am trying to scrape the text only from body using python Scrapy, but haven't had any luck yet.

Wishing some scholars might be able to help me here scraping all the text from the <body> tag.

Solution

  • Scrapy uses XPath notation to extract parts of a HTML document. So, have you tried just using the /html/body path to extract <body>? (assuming it's nested in <html>). It might be even simpler to use the //body selector:

    x.select("//body").extract()    # extract body

    You can find more information about the selectors Scrapy provides here.