Search code examples
pythonweb-scrapingbeautifulsoupfindall

unknown error in beautifulsoup web scraping using python

<div id="browse_in_widget">
<span id="browse_in_breadcrumb" style="width: 583px;">
<div class="seo_itemscope" itemtype="http://data-vocabulary.org/Breadcrumb" itemscope="">
<a itemprop="url" href="/search/"> Arabian Area</a>
<span class="seo_itemprop-title" itemprop="title">Arabian Area</span>
>
</div>
<div class="seo_itemscope" itemtype="http://data-vocabulary.org/Breadcrumb" itemscope="">
<a itemprop="url" href="/property-for-rent/home/"> Phase 2 </a>
<span class="seo_itemprop-title" itemprop="title">Phase 2 </span>
>
</div>
<div class="seo_itemscope" itemtype="http://data-vocabulary.org/Breadcrumb" itemscope="">
<a itemprop="url" href="/property-for-rent/residential/"> Residential Units for Rent </a>
<span class="seo_itemprop-title" itemprop="title">Residential Units for Rent</span>
>
</div>
<div class="seo_itemscope" itemtype="http://data-vocabulary.org/Breadcrumb" itemscope="">
<a itemprop="url" href="/property-for-rent/residential/apartmentflat/"> Apartment/Flat for Rent </a>
<span class="seo_itemprop-title" itemprop="title">Apartment/Flat for Rent</span>
>
</div>
<strong class="seo_itemprop-title" itemprop="title">Details</strong>
</span>
</div>

I want to get

['Arabian Area', 'Phase 2', 'Residential Units for Rent','Apartment/Flat for Rent']

I am trying to use the following code using beautiful 4 python

try:

        Type = [str(Area.text) for Area in soup.find_all("span", {"class" : "seo_itemscope"})]

        Area=' , '.join(Area)
        print Area


    except StandardError as e:
        Area="Error was {0}".format(e)
        print Area

All i want is to get the desired output in a list but there seems to be some problem. I am not getting any print. What can be the problem?

Thank you!

Solution

  • The first problem is that you are looking for span elements with seo_itemscope class which don't exist. Use seo_itemprop-title if you are looking for the titles:

    Type = [item.get_text() for item in soup.find_all("span", {"class": "seo_itemprop-title"})]

    The other problem is here:

    Area=' , '.join(Area)

    You meant to join items of the Type list instead:

    Area = ' , '.join(Type)

    And, it is not a good idea to catch the StandardError - it is too broad of an exception and actually is close to having a bare except clause. You should catch more specific exceptions, see: