Search code examples
pythondjangoamazon-web-servicescron

How to run .py files as cronjob in aws?


I have a Django application which it's deployed to Amazon Elastic Beanstalk(Python 3.7 running on 64bit Amazon Linux 2/3.1.1). I have been trying to run a .py file as a cronjob that works at 4 a.m every day in AWS and I have created a .config file into .ebextensions for that such as below.

cron.config file:

files:
    "/etc/cron.d/cron_process":
        mode: "000644"
        owner: root
        group: root
        content: |
            0 4 * * * root /usr/local/bin/task_process.sh

    "/usr/local/bin/task_process.sh":
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/bin/bash

            date > /tmp/date
            source /var/app/venv/staging-LQM1lest/bin/activate
            cd /var/app/current
            python Bot/run_spiders.py
            exit 0

commands:
    remove_old_cron:
        command: "rm -f /etc/cron.d/cron_process.bak"

run_spiders.py file:

from first_bot.start import startallSpiders
from .models import Spiders
import importlib

test = Spiders.objects.get(id=1)
test.spider_name = "nigdehalk"
test.save()

I'm trying to change an attribute of one of my objects for testing but it didn't work. Am I missing something? How can I create this cronjob?


Solution

  • Based on the comments.

    The issue was caused by failure of task_process.sh. Normally, cron will not show error messages if task_process.sh fail. To help troubleshoot the issue, you can redirect all output into a text file using:

    exec &>> /tmp/cron_capture_log.txt
    

    Subsequently, the EB config file could be:

    files:
        "/etc/cron.d/cron_process":
            mode: "000644"
            owner: root
            group: root
            content: |
                0 4 * * * root /usr/local/bin/task_process.sh
    
        "/usr/local/bin/task_process.sh":
            mode: "000755"
            owner: root
            group: root
            content: |
                #!/bin/bash
    
                exec &>> /tmp/cron_capture_log.txt
    
                date > /tmp/date
                source /var/app/venv/staging-LQM1lest/bin/activate
                cd /var/app/current
                python Bot/run_spiders.py
                exit 0
    
    commands:
        remove_old_cron:
            command: "rm -f /etc/cron.d/cron_process.bak"
    

    The above helps to identify issues with task_process.sh when running in cron.