Search code examples
beautifulsoupspaceplaintext

How to remove redundant space in BeautifulSoup output


I intend to scrape a website using BeautifulSoup. I'm working on the following HTML :

html = 
<div id="article-body" itemprop="articleBody">
<p>
    <span class="quote down bgQuote" data-channel="/quotes/zigman/170324/composite" data-bgformat="">
    <a class="qt-chip trackable" data-fancyid="XNYSStockSLB" href="/investing/stock/slb?mod=MW_story_quote" data-track-mod="MW_story_quote">
    SLB,
    <span class="bgPercentChange">-3.04%</span>
    </a>
    </span>
    reported late Thursday
    <a href="/story/schlumberger-profit-falls-sharply-2016-10-20-174854654" class="icon none">higher third-quarter profit that beat targets and sales only slightly below estimates</a>
    . Schlumberger’s results came a day after rival Halliburton Co.
    <span class="quote down bgQuote" data-channel="/quotes/zigman/228631/composite" data-bgformat="">
    <a class="qt-chip trackable" data-fancyid="XNYSStockHAL" href="/investing/stock/hal?mod=MW_story_quote" data-track-mod="MW_story_quote">
    HAL,
    <span class="bgPercentChange">-0.66%</span>
    </a> """

I want to get a plain text without any redundant space,I followed the answer by Twig but SLB and -3.04% and also HAL and -0.66% are still placed in different lines.My favorable output would be like :

 SLB, -3.04%  reported late Thursday higher third-quarter profit that beat targets and sales only slightly below estimates. Schlumberger’s results came a day after rival Halliburton Co. HAL, -0.66%  also posted higher-than-expected profit.

It is my code:

import urllib2
from bs4 import BeautifulSoup
import re
newsText = soap(html)
text = list(newsText.stripped_strings)
finalText = "\n\n".join(text) if descriptions else ""
re.sub(r'[\ \n]{2,}', '', finalText)
print finalText

I am very thankful in advance.


Solution

  • soup = BeautifulSoup(html, 'lxml')
    text = soup.get_text(strip=True, separator='  ')
    print(text)
    

    out:

    SLB,  -3.04%  reported late Thursday  higher third-quarter profit that beat targets and sales only slightly below estimates  . Schlumberger’s results came a day after rival Halliburton Co.  HAL,  -0.66%