Search code examples
pythonhtmlparsingbeautifulsoupextract

Extract a specific header from HTML using beautiful soup


This is the patent example I am using https://patents.google.com/patent/EP1208209A1/en?oq=medicinal+chemistry . Below is the code I used. I want the code to display only the cited by (3) count so I know how many times this patent was cited.How can I get the output to display the cited by count as 3 only? Kindly help!

 
soup = BeautifulSoup(patent, 'html.parser')
cited_section =soup.findAll({"h2":"Cited By"})

print(cited_section)
Output I get is [<h2>Info</h2>, <h2>Links</h2>, <h2>Images</h2>, <h2>Classifications</h2>, <h2>Abstract</h2>, <h2>Description</h2>, <h2>Claims (<span itemprop="count">57</span>)</h2>, <h2>Priority Applications (5)</h2>, <h2>Applications Claiming Priority (1)</h2>, <h2>Related Parent Applications (1)</h2>, <h2>Publications (2)</h2>, <h2>ID=38925605</h2>, <h2>Family Applications (1)</h2>, <h2>Country Status (1)</h2>, <h2>Cited By (3)</h2>, <h2>Families Citing this family (12)</h2>, <h2>Citations (306)</h2>, <h2>Patent Citations (348)</h2>, <h2>Non-Patent Citations (23)</h2>, <h2>Cited By (4)</h2>, <h2>Also Published As</h2>, <h2>Similar Documents</h2>, <h2>Legal Events</h2>]````

Solution

  • The number of citations is created dynamically via JavaScript. But you can count number of elements with itemprop="forwardReferencesFamily" to get the count. For example:

    import requests
    from bs4 import BeautifulSoup
    
    
    url = 'https://patents.google.com/patent/EP1208209A1/en?oq=medicinal+chemistry'
    soup = BeautifulSoup(requests.get(url).content, 'html.parser')
    
    print(len(soup.select('tr[itemprop="forwardReferencesFamily"]')))
    

    Prints:

    4