I have a python script running in background with nohup ./myprogram.py 1>console.out &. The program is constantly logging to some log file and the processing is long. After running for 2 days(Saturday and Sunday) I see
# myprogram and myprogram2 are both running in background
# myprogram2 clearly has finished
[1] + 25159 exit 1 nohup ./myprogram.py 1>console.out &
[2] + 25442 done nohup ./myprogram2.py 1>console2.out &
The log for myprogram
2016-05-27 16:55:06 - sources.edf - INFO - processing day 1 ...
2016-05-27 16:55:06 - sources.edf - INFO - processing day 2 ...
...
2016-05-27 16:55:06 - sources.edf - INFO - processing day n ...
and stopped(there should be n + 1 and more).
Sadly I just overwrote on console.out already(so dump to overwrite on it before I even had a look... but I was continuing from day n and seems the program can be run without any error/exception)
I know this description is a bit or just too vague to point out any reason for this exit. I just need to know some clue about this. I am not completely new to this but I do lack experience. So any possible guess is appreciated.
The simplified source code:
import os
import sys
import logging
import logging.config as lconfig
from optparse import OptionParser
from contextlib import closing
from datetime import datetime, timedelta
from collections import defaultdict
import psycopg2
from configobj import ConfigObj
if __name__ == "__main__":
## setup optparser and parse argvs and return opts and args
conf = ConfigObj(opts.config, list_values=False)[args[0]]
__import__(conf["module"])
## myprogram and myprogram2 is running the same source code
## same module. only the data is different
## source will provide Mapper and iterator as APIs
source = sys.modules[conf["module"]]
## extract start and end date, configure log
# set up regions info and mapper
## connect to db and read countries and exchange list
with closing(psycopg2.connect(conf["dsn"])) as db:
cursor = db.cursor()
regions = source.setup_region(conf['Regions'], cursor)
## find all wanted exchanges: (exch, region)
exchanges = source.setup_exchanges(conf['Exchanges'], cursor)
mapper = source.Mapper(exchanges, cursor, conf)
iterator = source.iterator(conf, start, end)
logger = logging.getLogger()
logger.info('START')
with regions:
for filename, raw_rec in iterator:
logger.info('Processing file {0}'.format(filename)
try:
record = source.Record(filename, raw_rec)
except Exception as e:
logger.warn("record parsing error: %s" % e)
continue
stks = mapper.find(record)
if not stks:
continue
regs = defaultdict(set)
for stk in stks:
regnm = exchanges[stk[2]]
regs[regnm].add(stk)
for reg,secs in regs.iteritems():
info = regions[reg]
outf = info.get_file(record.get_tm())
source.output(outf, record, secs, info.tz, conf)
logger.info('END')
The log just stopped as Prccessing some file...
Database connection can drop at any moment and your code does not deal with this issue. Also computers do need to to restart from time to time, like power-cuts, security updates.
Obviously that there are other reasons for which your code could break and that's why you need to make it run as a service in case you want to make it reliable.
Learn how to run this python script as a systemd service and configure this service to restart automatically if it fails. This would prevent it from stopping even if there is a bug that crash it. Systemd will restart it if it does crash.
There are lots of resources documenting this but you can start with https://learn.adafruit.com/running-programs-automatically-on-your-tiny-computer/systemd-writing-and-enabling-a-service