I've posted about this question before, and I thought I had it solved, but unfortunately I still can't figure out what's going on.
In a nutshell: I have one JSON file in an s3 bucket which is updated daily. My crontab script (set to run once daily) downloads the file from my s3 bucket to my local directory, overwriting the existing file on my local directory. The script is run locally, on Mac terminal. The intention is to run the script once a day so that the file is constantly overwritten and updated.
I have verified that the JSON file in the s3 bucket is being continually updated, so I don't think that's the problem.
Here's the command, on crontab:
03 23 * * * /Library/Frameworks/Python.framework/Versions/3.7/bin/aws s3 cp s3://sfbucket.bucket/sf_events.json /Users/Documents/TownSounds_Javascript/data/sf_events.json >> /Users/Documents/logs3.txt 2>&1
The above crontab script is failing to run. The only way I can get it to run is by editing the crontab script once, manually: * * * * *
Whereupon it runs one time, and then reverts to not running at all.
When I check the logs3.txt, the date modified does not appear to be updating, which is also an indication of it's failure to run.
I've further investigated, and created a test crontab script:
* * * * * echo `date` >> /Users/Documents/testCrontab.txt
And it works fine, outputting exactly as expected.
Can anyone tell me why the crontab script for the s3 file may be failing to run? Do I need to be logged in to s3 for the script to be able to do it's thing and pull the file? Amazon s3 automatically logs me out after a period of inactivity, so this seems infeasible.
I've read that aws s3 sync might be what I'm looking for?
There are many reasons why an AWS CLI command may fail (in this case, aws s3 cp)... you (or the script that is running that command, e.g. crontab ) need to have appropriate credentials to interact with your aws resources (see: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). You need to check that the ACCESS_KEY & SECRET_KEY are available either as an ENV variable or you can provide it directly to the aws command (see: passing access and secret key aws cli)