Im using python 3.8x to try and scrape a randomly generated sentence from this website. https://randomwordgenerator.com/sentence.php Except when I read it, the generated sentence isn't in the HTML. Can any one help me find a way to scrape the generated sentence? I found the HTML tags when the sentence is generated but it doesn't generate when I request.
Here is my code.
random_sentence_webpage = 'https://randomwordgenerator.com/sentence.php'
# The HTML tag for the generated sentence
start_marker = '"support-sentence">'
end_marker = '</span>'
from urllib.request import urlopen, Request
headers = {'User-Agent': 'Chrome/81.0.4044.129'}
reg_url = random_sentence_webpage
req = Request(url=reg_url, headers=headers)
html = urlopen(req).read()
html_text = html.decode('utf-8', 'backslashreplace')
starting_position = html_text.find(start_marker)
end_position = html_text.find(end_marker,starting_position)
random_generated_sentence = html_text[starting_position + len(start_marker):end_position]
print(random_generated_sentence)
You will find more details here Using python Requests with js pages
but the short solution is using requests_html:
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://randomwordgenerator.com/sentence.php')
r.html.render()
print(r.html.find(".support-sentence")[0].text)
outputs
Having no hair made him look even hairier.