Lately I have been doing some interactive work in Python.
My setup is an IPython notebook running on a server that uses a grid engine to manage jobs.
Today I was trying to get an IPython cluster going following an example posted here that uses subprocess.Popen to start a the cluster.
I couldn't get the example to work so I tried opening up the IPython/Jupyter terminal emulator and typing in the ipcluster start command and the cluster started right up!
After playing around with things for a while I realized that if I typed env in the terminal emulator I got a different list of environment variables than when I looked at the os.environ variable in Python. The source of the problem seemed to be that the PATH variables were different.
Now I know that I can change the PATH variable in os.environ, but I'm wondering why it is different in the first place? I know very little about environment variables, so this may be a stupid question, but I would have assumed that a terminal emulator and notebook running on the exact same node in the exact same IPython notebook server would have had the exact same environment variables.
Any insight on why the environment variables in the terminal and notebook might be different will be greatly appreciated.
Update: In case it matters, the server I am working on uses the Univa Grid Engine. Also I have noticed that it seems to make a difference whether I use qrsh or qsub to start the notebook server.
Previously I had been using qsub, but by starting the notebook server with qrsh I eliminate many of the differences between env and os.environ. There are still differences, but much fewer. Still not sure what any of that means though:)
As per manual page for qsub, qsh, qrsh, to propagate current shell environment to the job use the -V option:
-V Available for qsub, qsh, qrsh with command and qalter.
Specifies that all environment variables active within the qsub utility be exported to the context of the job.
All environment variables specified with -v, -V or the DISPLAY variable provided with -display will be exported to the defined JSV instances only optionally when this is
requested explicitly during the job submission verification. (see -jsv option above or find more information concerning JSV in jsv(1))