Find Specific Text Within HTML Tag in Python

Question

I've tried a million different ways to parse out the zestimate, but have yet to be successful.

here's the html tag with the zestimate info:

<span>
  <span tabindex="0" role="button">
    <span class="sc-bGbJRg iiEDXU ds-dashed-underline">
      Zestimate
    <sup>®</sup>
    </span>
  </span>
  :&nbsp;
  <span>$331,425</span>
</span>

Honestly I thought this would get me close, but I get an empty list:

link = 'https://www.zillow.com/homedetails/1404-Clearwing-Cir-Georgetown-TX-78626/121721750_zpid/'
searched_word = '<span class="sc-bGbJRg iiEDXU ds-dashed-underline">Zestimate<sup>®</sup></span>'
test_page = requests.Session().get(link, headers=req_headers)
test_soup = BeautifulSoup(test_page.content, 'lxml')
results = test_soup('span',string='searched_word')
print(results)[0]

Answer 1

To get correct HTML from the site, add User-Agent header to request.

For example:

import requests
from bs4 import BeautifulSoup


url = 'https://www.zillow.com/homedetails/1404-Clearwing-Cir-Georgetown-TX-78626/121721750_zpid/'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}
soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')

home_value = soup.select_one('h4:contains("Home value")').find_next('p').get_text(strip=True)
print(home_value)

Prints:

$331,425