I have a python script that connects to Redshift, executes a series of SQL commands, and generates a new derived table.
But for the life of me, I can't figure out a way to have it automatically run every day.
I've tried AWS Data Pipeline but my shell script won't run the first copy statement. I can't get Lambda or Glue to work because my company's IAM policies are restrictive. Airflow seems like overkill to just run a single python script daily.
Any suggestions for services to look into?
use a cron job on an ec2 instance or set up a scheduled event to invoke your aws python lambda function http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html