We have observed a long running Python 2.7.5 process leaking memory. I tried to inject some code into the process using gdb, to see what's happening inside. The gdb command looks like:
gdb -p $pid -batch -eval-command='call PyGILState_Ensure()' -eval-command='call PyRun_SimpleString("exec(open(\"/path/to/code\").read())")' -eval-command='call PyGILState_Release($1)'
I printed gc.get_objects() information into a file and took 2 snapshots, but failed to find obvious object size increase. But after calling gc.collect() in the injected code, a dramatic memory decrease is observed. So I tried to print the return value following:
gc.isenabled()
gc.get_count()
gc.get_threshold()
which gives me:
False
(56107, 0, 0)
(700, 10, 10)
This obviously shows that gc is disabled, but I'm sure non of our code does it explicitly. And the same code is running on hundreds of servers but only one has this problem.
Does anyone have any idea what might be the cause, or what to check to find the root cause?
Just found a very possible cause in python issue tracker. See comment by torsten in https://bugs.python.org/issue1336
Among other issues (like a hang for still unknown reasons when calling subprocess.Popen) I have also seen gc getting disabled for a long running process which has multiple threads and starts processes via subprocess.Popen from different threads.
In python 2.7, when calling subprocess in multiple threads, there's possiblility that gc will be disabled.
Post this info here in case someone else also run into this problem.