python does not release filehandles to logfile

Question

I have an application which has to run a number of simulation runs. I want to setup a logging mechanisme where all logrecords are logged in a general.log, and all logs for a simulation run go to run00001.log, .... For this I have defined a class Run. in the __init__() a new filehandle is added for the runlog.

The problem is that the logfiles for the runs never get released, so after a number of runs the available handles are exhausted and the run crashes.

I've set up some routines to test this as follows

main routine

import Model
try:
    myrun = Model.Run('20130315150340_run_49295')
    ha = raw_input('enter')
    myrun.log.info("some info")
except:
    traceback.print_exc(file=sys.stdout)

ha = raw_input('enter3')

The class Run is defined in module Model as follows

import logging
class Run(object):

    """ Implements the functionality of a single run. """
    def __init__(self, runid):
        self.logdir="."
        self.runid          = runid
        self.logFile        = os.path.join(self.logdir , self.runid + '.log')
        self.log            = logging.getLogger('Run'+self.runid)
        myformatter         = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
        myhandler      = logging.FileHandler(self.logFile)
        myhandler.setLevel(logging.INFO)
        myhandler.setFormatter(myformatter)
        self.log.addHandler(myhandler) 

Then I use the program process explorer to follow the filehandlers. And I see the runlogs appear, but never disappear.

Is there a way I can force this?

Answer 1

You need to call .close() on the filehandler.

When your Run class completes, call:

handlers = self.log.handlers[:]
for handler in handlers:
    self.log.removeHandler(handler)
    handler.close()

A file handler will automatically re-open the configured filename every time a new log message arrives, so calling handler.close() may sometimes appear futile. Removing the handler from the logger stops future log records from being sent to it; in the above code we do this first, to avoid an untimely log message from another thread reopening the handler.

Another answer here suggest you use logging.shutdown(). However, all that logging.shutdown() does is call handler.flush() and handler.close(), and I'd not recommend using it. It leaves the logging module in a state where you can't use logging.shutdown() again, not reliably.