I have put together a ETL job written in Python. I have to run it through different environments (LOCAL, DEV, TST, PROD) so I need to find a way to set the job schedule using environment variables so that I can set different schedules for each environment without touching the code from one environment to another. Today I am using apscheduler. I have something like this in the code:
from apscheduler.schedulers.blocking import BlockingScheduler
def etl_job():
sched = BlockingScheduler(daemon=True)
sched.add_job(etl_job,'interval',minutes=7)
sched.start()
Does anyone know now to solve this? Thanks!
Here how I've solved this one:
Environment variable (json format)
schedule='{"jobsSchedule": [{"job_name": "job_1","schedule": "*/1 * * * *"},{"job_name":"job_2","schedule": "*/2 * * * *"}]}'
Read the env variable
jobCrontabSchedule = json.loads(os.getenv('schedule'))
Schedule each one of the jobs
for job in jobCrontabSchedule['jobsSchedule']:
print ("Scheduling the job: "+job['job_name']+" with the schedule: "+job['schedule'])
sched.add_job(globals()[job['job_name']], CronTrigger.from_crontab(job['schedule']))