I would like to know how to automatically execute a code in Django within a defined period of time. I'm building a web crawler that collects information and stores it in a JSON file and a function that reads the file and stores the file information in a SQLite database. This information is rendered and visible on my website. At the moment I have to run the crawler and the function that saves the data in a database with a click on a button, but that is very ineffective. It would be better if the database information were updated automatically, about every 6 hours (of course, only if the server is running).
If you want to keep the code on your webserver and you have the permissions then a CRON job (Linux) or Scheduled Task (Windows) will do what you want. Set your cron job to run every six hours and to call your Django script.
You can run the script as a Django command line, eg manage.py mycommand or run as a stand-alone PY file where you include the libs and run django.setup()
Or if you want the crawler to be run from within the context of your webserver then your cron job can initiate a GET request using HTTPIE. That gives you a nice API approach.
When I run this I have a flag in a database (a model called Job) that records what is running just in case my scheduled task is already running when the cron gets called, I have some very long-running asynchronous tasks and I tend to use cron rather than Celery.
If you are running your site on AWS or GCP then both have a cron equivalent. Eg you could create a GCP cloud scheduler link that fires off a GET to your webserver that triggers the code to run the crawler.