Search code examples
javapythonsocketsforkpyramid

Pyramid, fork, Java and sockets

A complicated situation:

  1. A Pyramid app needs to start/restart some Java processes (e.g. Jetty) sometimes (e.g. on receiving new WAR files for Jetty, etc, etc).

  2. That Jetty process is expected to continue working after Pyramid process (pserve) quits or between Pyramid restarts.

  3. Jetty process is started via subprocess.Popen + shell script. Naturally, being a child process, it inherits file descriptors, including sockets.

  4. Now if Pyramid needs to restart while Jetty is still running, it can't do so as child Jetty process still has the socket bound to Pyramid's main address/port.

Solution so far:

  1. For restarting Java/Jetty, do a fork

  2. in a child process close every file descriptor > 3 and < resource.RLIMIT_NOFILE.

  3. do sys.exit in a child process after starting/restarting Jetty.

Neat, right?

No.

It's complicated and kludgy. Is there a simpler way of avoiding this "child process inheriting sockets" problem?

Plus I have services (like APScheduler, etc) running and that requires elaborate shutdowns of all the running services (in a child process of course) or else they raise exceptions on sys.exit.

Solution

  • Is there a simpler way of avoiding this "child process inheriting sockets" problem?

    Yes, just pass close_fds=True to the Popen constructor or wrapper function. This closes all fds except 0, 1, and 2.*

    If you need to keep some different set alive instead of 0, 1, and 2, use pass_fds=[0, 2, special_file.fileno()] instead of close_fds=True. But hopefully you don't need that.**

    * I'm assuming you don't care about Windows portability—after all, you're already using os.fork. If I'm wrong, close_fds=True won't work on non-POSIX systems with redirected stdio handles, so if you need both, you'll need a more complex solution.

    ** If you do: pass_fds requires 3.2+, or a backport of the 3.2+ subprocess module. I'm pretty sure you don't have 3.2+, because if you did, close_fds=True would already be the default behavior on POSIX systems, and you wouldn't have this problem in the first place. One common reason to need it for web services is when you don't have permissions to bind port 80, so you inherit a socket from a suid program, and need to pass that socket to your own child.