First-time user of AWS/EC2/ubuntu here. I'm having an issue running a python script (a web scraper, using chromedriver, selenium, beautifulsoup, etc) on ubuntu server using crontab. This is an EC2 instance on AWS. I've gotten crontab to work as a simple test, creating a simple txt file that reads 'hello'. However, this current cronjob keeps failing:
It's pretty simple, just executes one script at 12:17 PM daily:
ubuntu@ip-172-31-31-200:~$ crontab -l
17 12 * * * python3 BandsInTown_Scraper_SF.py
The job does not execute. My crontab status reads:
ubuntu@ip-172-31-31-200:~$ sudo service cron status
● cron.service - Regular background program processing daemon
Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2019-11-02 00:18:36 UTC; 1 weeks 0 days ago
Docs: man:cron(8)
Main PID: 830 (cron)
Tasks: 1 (limit: 1152)
CGroup: /system.slice/cron.service
└─830 /usr/sbin/cron -f
Nov 09 16:17:01 ip-172-31-31-200 CRON[27857]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 09 16:17:01 ip-172-31-31-200 CRON[27858]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 09 16:17:01 ip-172-31-31-200 CRON[27857]: pam_unix(cron:session): session closed for user root
Nov 09 17:17:01 ip-172-31-31-200 CRON[27908]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 09 17:17:01 ip-172-31-31-200 CRON[27909]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 09 17:17:01 ip-172-31-31-200 CRON[27908]: pam_unix(cron:session): session closed for user root
Nov 09 18:15:01 ip-172-31-31-200 cron[830]: (ubuntu) RELOAD (crontabs/ubuntu)
Nov 09 18:17:01 ip-172-31-31-200 CRON[28242]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 09 18:17:01 ip-172-31-31-200 CRON[28243]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 09 18:17:01 ip-172-31-31-200 CRON[28242]: pam_unix(cron:session): session closed for user root
And, when I dig into the log of a past failed cron job, I get:
Nov 5 02:17:01 ip-172-31-31-200 CRON[14524]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 5 02:26:58 ip-172-31-31-200 systemd[1]: Starting Message of the Day...
Nov 5 02:26:58 ip-172-31-31-200 50-motd-news[14532]: * Kata Containers are now fully integrated in Charmed Kubernetes 1.16!
Nov 5 02:26:58 ip-172-31-31-200 50-motd-news[14532]: Yes, charms take the Krazy out of K8s Kata Kluster Konstruction.
Nov 5 02:26:58 ip-172-31-31-200 50-motd-news[14532]: https://ubuntu.com/kubernetes/docs/release-notes
Nov 5 02:26:58 ip-172-31-31-200 systemd[1]: Started Message of the Day.
Nov 5 02:36:28 ip-172-31-31-200 systemd-timesyncd[577]: Network configuration changed, trying to establish connection.
Nov 5 02:36:28 ip-172-31-31-200 systemd-timesyncd[577]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 5 03:06:28 ip-172-31-31-200 systemd-timesyncd[577]: Network configuration changed, trying to establish connection.
Nov 5 03:06:28 ip-172-31-31-200 systemd-timesyncd[577]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 5 03:17:01 ip-172-31-31-200 CRON[14621]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 5 03:36:28 ip-172-31-31-200 systemd-timesyncd[577]: Network configuration changed, trying to establish connection.
Nov 5 03:36:28 ip-172-31-31-200 systemd-timesyncd[577]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 5 03:39:01 ip-172-31-31-200 CRON[14646]: (root) CMD ( test -x /etc/cron.daily/popularity-contest && /etc/cron.daily/popularity-contest --crond)
Nov 5 04:06:27 ip-172-31-31-200 systemd-timesyncd[577]: Network configuration changed, trying to establish connection.
Nov 5 04:06:27 ip-172-31-31-200 systemd-timesyncd[577]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 5 04:17:01 ip-172-31-31-200 CRON[14675]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 5 04:27:25 ip-172-31-31-200 systemd[1]: Started Session 144 of user ubuntu.
Nov 5 04:36:27 ip-172-31-31-200 systemd-timesyncd[577]: Network configuration changed, trying to establish connection.
Nov 5 04:36:27 ip-172-31-31-200 systemd-timesyncd[577]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 5 04:41:04 ip-172-31-31-200 systemd[1]: Started Session 145 of user ubuntu.
Nov 5 04:41:09 ip-172-31-31-200 crontab[14897]: (ubuntu) LIST (ubuntu)
The line Network configuration changed, trying to establish connection. is concerning and wondering if that's the root cause of this. Does ubuntu server need to be up and running in order from crontab to work? I ask because when I log in to ubuntu on terminal, and then leave it running for awhile and my computer goes to sleep, I get the following error:
ubuntu@ip-172-31-31-200:~/bandsintown$ packet_write_wait: Connection to 3.132.213.71: Broken pipe
Is this the root cause of my issue? I thought the whole point of running code on AWS was that it could run independently of my local machine (i.e. if the internet goes out, if the terminal logs out of Ubuntu, etc).
None of the answers provided worked, however, this did work:
17 12 * * * /usr/bin/python3.6 /home/ubuntu/bandsintown/BandsInTown_Scraper_SF.py