Search code examples
pythonapiamazon-s3amazon-ec2cron

Crontab not executing Python3 script on AWS EC2 Linux instance

I can't get a basic python3 script to run using cron on my AWS EC2 Linux instance.

Project structure:

ec2-user
  └─ log.txt
  └─ scripts
       └─ test.py

Python script:

#!/usr/bin/python3
import pandas as pd
import requests
response = requests.get(url = 'https://api.coindesk.com/v1/bpi/currentprice.json')
response = response.json()
response = pd.DataFrame.from_dict(data = response['bpi'])
response.to_csv('s3://my-bucket/test.csv')

Cron:

* * * * * /usr/bin/python3 /home/ec2-user/scripts/test.py 2>&1 >/home/ec2-user/log.txt

Cron appears to be active based on the below output after running sudo systemctl status crond:

Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-07-29 17:33:54 UTC; 23h ago
Main PID: 3064 (crond)
CGroup: /system.slice/crond.service
        └─3064 /usr/sbin/crond -n

I don't think the problem is with the python script because I can successfully run it in my bash terminal with python3 ./scripts/test.py.

Here's what I've tried:

  • Various version of shebang line in python script
  • Change paths from relative to absolute
  • Verified location of python3 is correct
############### EDIT 1 ###############

Checking /var/log/messages gives me this:

systemd: Created slice User Slice of root.
systemd: Started Session 23 of user root.
systemd: Removed slice User Slice of root.

Checking /var/log/cron gives me this:

CROND[6260]: (root) CMD (/usr/bin/python3 /home/ec2-user/scripts/test.py 2>&1 >/home/ec2-user/log.txt)
Solution

  • The issue was me not understanding at what level cron was running. I set everything up at the user level, but cron was running at the root level.

    While looking into /var/log that @link89 suggested, I noticed the logs referenced root. I switched from ec2-user to root by entering sudo su in my bash terminal. Then I ran /usr/bin/python3 /home/ec2-user/scripts/test.py in the bash terminal which returned a ModuleNotFound error.

    I installed all the required python modules at the root level with sudo pip3 install <module-name>. I also had to configure my AWS CLI at the root level with aws configure. The cron job ran the python script successfully after these changes.