This is my first time developing a website and I am coming across some issues. I have some python code that scrapes some data with beautifulsoup4 and displays the number on my site with flask. However, I found that my site does not automatically update the values at all, and rather only updates when I manually make my host reload.
How can I make it so that my python script "re-scrapes" every time a visitor visits my site, or just every 5 minutes or so? Any help would be greatly appreciated! emphasized text Host- Pythonanywhere
Here is my current backend python code:
import bs4 as bs
import urllib.request
from flask import Flask, render_template
app = Flask(__name__)
link = urllib.request.urlopen('https://www.health.pa.gov/topics/disease/coronavirus/Pages/Cases.aspx')
soup = bs.BeautifulSoup(link, 'lxml')
body = soup.find('body') # get the body so you can do soup.find_all() inside it
tables = soup.find_all('table')
for table in tables:
table_rows = table.find_all('tr')
for tr in table_rows:
td = tr.find_all('td')
row = [i.text for i in td]
if row.count('Bucks') > 0:
print(row[1])
# Bucksnum shows the amount of cases in bucks county,
bucksnum = str(row[1])
data = bucksnum
# this is the part that connects the flask file to the html file
@app.route("/")
def home():
return render_template("template.html", data=data)
@app.route("/")
def index():
return bucksnum
if __name__ == '__main__':
app.run(host='0.0.0.0')
index()
You would need to use schedulers, check this thread that discusses a similar issue, you can use it to call a certain function that updates your data every interval of time.