I am running a detached child program in background from my parent program. After i exited the parent program, i would expect the child program to continuing running and logging into OUTPUT_PATH. And indeed, i can see the log file updating. However as i was trying to find the PID from ps aux i can't find it. can anyone explain this behavior? what am i doing wrong?
shellCommand = "nohup python PYTHON_PROGRAM ARGS >OUTPUT_PATH 2>&1 &"
subprocess.Popen(shellCommand, shell=True, preexec_fn=os.setpgrp)
OK, this is getting too big for comments. By running ps -fwp $(pgrep -f PYTHON_PROGRAM), we've found the process now. :) But its PID does not match the one reported by Popen.pid. This would be down to shell instance that was called since you've used shell=True. First fork was to call shell, the second was for your script. Actually, this is documented in the link mention above:
Note that if you set the shell argument to True, this is the process ID of the spawned shell.
But see the NOTE bellow.
Which brings us to the "more orthodox way". Where we're entering possibly contested territory, different people, different ideas. Not as much the first one perhaps as it would be in line with the documentation to suggest not to use shell=True unless you really need to.
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
There is also another section on (security) implications of not heeding the recommendation.
So, compiling a list of arguments to run nohup with your script and handling the output redirection already through keyword arguments (stdout, stderr) of Popen would seem like a good course of action and would also get you a consistent PID.
This last step might attract most controversy: but you can actually daemonize process by means of python interfaces to corresponding syscalls. Well documented example would seem to grow in github (reached over one hop from a link in the PEP mentioned bellow).
or there is a library referred to from the PEP-3143 on the topic.
NOTE: That bit does not appear to be always true (calling of sh yes, but two PIDs no). At least on my system I've observed sh to exec the program called via -c (in itself) without forking. From few quick runs and traces this was the case at least if I did not mess with the stdin/-out/-err (i.e. no pipes or redirections), did not force subshell (...), or did not chain commands over ;. (The latter two are kind of obvious, former is as well once you realize how redirections are implemented). So at least for my shell I would dare to extrapolate and say that: It seems it does not fork, unless it must. Or even more simplified (and hence not entirely correct) statement would be: simple stuff won't fork.