I intend to scrape a website using BeautifulSoup. I'm working on the following HTML :
html =
<div id="article-body" itemprop="articleBody">
<p>
<span class="quote down bgQuote" data-channel="/quotes/zigman/170324/composite" data-bgformat="">
<a class="qt-chip trackable" data-fancyid="XNYSStockSLB" href="/investing/stock/slb?mod=MW_story_quote" data-track-mod="MW_story_quote">
SLB,
<span class="bgPercentChange">-3.04%</span>
</a>
</span>
reported late Thursday
<a href="/story/schlumberger-profit-falls-sharply-2016-10-20-174854654" class="icon none">higher third-quarter profit that beat targets and sales only slightly below estimates</a>
. Schlumberger’s results came a day after rival Halliburton Co.
<span class="quote down bgQuote" data-channel="/quotes/zigman/228631/composite" data-bgformat="">
<a class="qt-chip trackable" data-fancyid="XNYSStockHAL" href="/investing/stock/hal?mod=MW_story_quote" data-track-mod="MW_story_quote">
HAL,
<span class="bgPercentChange">-0.66%</span>
</a> """
I want to get a plain text without any redundant space,I followed the answer by Twig but SLB and -3.04% and also HAL and -0.66% are still placed in different lines.My favorable output would be like :
SLB, -3.04% reported late Thursday higher third-quarter profit that beat targets and sales only slightly below estimates. Schlumberger’s results came a day after rival Halliburton Co. HAL, -0.66% also posted higher-than-expected profit.
It is my code:
import urllib2
from bs4 import BeautifulSoup
import re
newsText = soap(html)
text = list(newsText.stripped_strings)
finalText = "\n\n".join(text) if descriptions else ""
re.sub(r'[\ \n]{2,}', '', finalText)
print finalText
I am very thankful in advance.
soup = BeautifulSoup(html, 'lxml')
text = soup.get_text(strip=True, separator=' ')
print(text)
out:
SLB, -3.04% reported late Thursday higher third-quarter profit that beat targets and sales only slightly below estimates . Schlumberger’s results came a day after rival Halliburton Co. HAL, -0.66%