Options for deploying Flask app that continuously web scrapes

Question

I coded a Flask web app that tracks prices on Amazon. Here is an outline of what it does:

• The user signs up and logs into the app
• They input the URL of the product they want to track on Amazon as well as their budget and email
• the app will web scrape the URL and retrieve the current price. Then it will compare it with the budget and perform some actions

The problem I'm having is that when I deploy the app on Heroku for example, I want the app to run the web scraping and price checking every day on the product so that it can notice any changes to the price.

Does anyone know how to do this? Would I need to write a separate python script and what kind of web services would I need? I would prefer them to not cost any money

Answer 1

Have you tried using Cron? There is no cost nor is there an installation necessary: it runs on Unix-like operating systems and works in virtual environments to run time based jobs. So you could set a cronjob to run a python script every hour or so; in your case the script would be one that scrapes the Amazon website for prices.

Read about Cron here: https://en.wikipedia.org/wiki/Cron

And here is a useful tutorial made by something of a Flask guru, Miguel Grinberg: Run your Flask Regularly Scheduled Jobs with Cron

You could also have a python script that always runs and only webscrapes after a certain time has passed , but this would (a) uses more CPU – which is bad if it's a pay as you go virtual machine – and (b) has a performance impact on your app.