How to return the whole line where a string match is found in a webpage using python request

Question

I am working with grabbing the line that matches for the strings I am searching in a webpage. I tried some approach but it reads and displayed everything. Below is the partial snippet.

import requests
url = "https://bscscan.com/address/0x88c20beda907dbc60c56b71b102a133c1b29b053#code"
queries = ["Website", "Telegram", "Submitted"]

r = requests.get(url)
for q in queries:
    q = q.lower()
    if q in r.text.lower():
        print(q, 'Found')
    else:
        print(q, 'Not Found')

Current Output:

    website Found
    telegram Found
    submitted Found

Wanted Output:

    Submitted Found - *Submitted for verification at BscScan.com on 2021-08-08
    Website Found - *Website: www.shibuttinu.com
    Telegram Found - *Telegram: https://t.me/Shibuttinu

Answer 1

requests is returning an html page which you have to parse with an html parser. One problem is that your target outputs are stuck in the middle of a long string which, after parsing, you have to extract using some string manipulation.

You can parse the html with either beautifulsoup with css selectors, or lxml with xpath:

First, using lxml:

import lxml.html as lh

doc = lh.fromstring(r.text)

loc = doc.xpath('//pre[@class="js-sourcecopyarea editor"]')[0]
targets = list(loc.itertext())[0].split('*')
for target in targets:
    for query in queries:
           if query in target:
                print(target)

With beautifulsoup:

from bs4 import BeautifulSoup as bs

soup = bs(r.text,'lxml')

pre = soup.select_one('pre.js-sourcecopyarea.editor')
ss = (list(pre.stripped_strings)[0]).split('*')
for s in ss:
       for query in queries:
            if query in s:
                print(s)

Output, in either case:

Submitted for verification at BscScan.com on 2021-08-08

Website: www.shibuttinu.com
 
Telegram: https://t.me/Shibuttinu