Python : http.server.HTTPServer : How to close ALL opened files?

Question

So basically, I am making an HTTP webhooks server in Python 3 and wanted to add a restart function because shell access is very limited on the server it will be running on.

I found this snippet somewhere on Stack Overflow earlier:

def restart_program():
    """Restarts the current program, with file objects and descriptors
       cleanup
    """

    try:
        p = psutil.Process(os.getpid())
        fds = p.open_files() + p.connections()
        print (fds)
        for handler in fds:
            os.close(handler.fd)
    except Exception as e:
        logging.error(e)

    python = sys.executable
    os.execl(python, python, *sys.argv)

For the most part, it works, but I wanted to make sure so I ran a few tests with lsof and found that every time I restarted the server, two more lines (files) were added to the list of open files:

python3 13923 darwin    5u  systm 0x18cd0c0bebdcbfd7        0t0            [ctl com.apple.netsrc id 9 unit 36]
python3 13923 darwin    6u   unix 0x18cd0c0beb8fc95f        0t0            ->0x18cd0c0beb8fbcdf

(the adresses varying each restart)

These are only present when I initiate httpd = ThreadingSimpleServer((host, port), Handler). But even after I call httpd.server_close() these open files persist and psutil doesn't seem to find them.

This isn't really required feature. If this proves to be too much overhead I can drop it, but right now I am only interested in why my code doesn't work and a solution for my own sanity.

Thanks in advance!

UPDATE:

Changing p.connections() to p.connections(kind='all') got me the unix type fd. Still not sure how to close the systm type fd. Turns out the unix fd had to do with DNS...

UPDATE:

Well, it looks like I found a solution, however messy it may be.

class MyFileHandler(object):
    """docstring for MyFileHandler."""
    def __init__(self, fd):
        super(MyFileHandler, self).__init__()
        self.fd = fd

def get_open_systm_files(pid=os.getpid()):
    proc = subprocess.Popen(['lsof', '-p', str(pid)], stdout=subprocess.PIPE)
    return [MyFileHandler(int(str(l).split(' ')[6][:-1])) for l in proc.stdout.readlines() if b'systm' in l]

def restart_program():
    """Restarts the current program, with file objects and descriptors
       cleanup
    """

    try:
        p = psutil.Process(os.getpid())
        fds = p.open_files() + p.connections()
        print (fds)
        for handler in fds:
            os.close(handler.fd)
    except Exception as e:
        logging.error(e)

    python = sys.executable
    os.execl(python, python, *sys.argv)

It's not pretty, but it works.

If anyone could shed some light on what actually is/was going on I would very much like to know.

Answer 1

Mmm that looks like a very hackish way to restart a process and a bad idea in general. What is your use case? Why do you want to restart a process to begin with? Regardless from your motivations, the usual way to interact with processes in that sense is via signals. I am not aware of signals designed specifically to restart a process though. What you usually want to do is terminate it (SIGTERM) and maybe have something like systemd or zdaemon which will automatically restart it. You can even write a signal handler to execute cleanup functions on SIGTERM, and that is the correct way to do cleaning up. You don't usually want to restart a process though, let alone do it from the app itself. That looks like a recipe for troubles.