Search code examples
pythonpywin32

How should I (or should I) finalize python in the presence of PyWin32?

We have a piece of code which uses PyWin32 and also occasionally calls Py_Finalize() and Py_Initialize() to reinitialize python. We've found a bug, in which a dynamically created class, class error, that is created from a string with PyRun_String() in pywintypes (PyWinTypesmodule.cpp, line 860), is not recreated after the reinitialize. This class ends up being used after the reinitialize, and throws a "NoneType is not callable" error when trying to call the function len().

We have observed that the error stops occurring if, immediately before calling Py_Finalize(), we call the PyWinTypes export PyWinGlobals_Free(), which is in the same source file linked above.

Elsewhere in the PyWin32 codebase (in dllmain.cpp), there is the following function, which is called when cleaning up after doing COM registration/unregestration in regsvr32.exe:

void PyCom_DLLReleaseRef(void)
{
    /*** NOTE: We no longer finalize Python EVER in the COM world
         see pycom-dev mailing list archives from April 2000 for why
    ***/
    // Must be thread-safe, although cant have the Python lock!
    // only needed when we finalize.
    //  CEnterLeaveFramework _celf;
    LONG cnt = InterlockedDecrement(&g_cLockCount);
    // Not optimal, but anything better is hard - g_cLockCount
    // could always transition 1->0->1 at some stage, screwing this
    // up.  Oh well...
    if (cnt==0) {
        // Send a quit message to the registered thread (if we have one)
        if (dwQuitThreadId)
            PostThreadMessage(dwQuitThreadId, WM_QUIT, 0, 0);
        /*** Old finalize code
             if (bDidInitPython) {
             PyEval_RestoreThread(ptsGlobal);
             PyWinGlobals_Free();
             FreeGatewayModule();
             Py_Finalize();

             bDidInitPython=FALSE;
             }
        ***/
    }
}

There are two notable things here:

  1. There is a mysterious comment at the top, implying that there is a reason for not finalizing Python in this case. Unfortunately the "pycom-dev" mailing list that it refers to no longer appears to exist. I think this is the right link for the thread in question, but pythonpros.com appears to be some sedo domain parking thing.

  2. The commented out "old finalize code" does call PyWinGlobals_Free() before finalizing python, which suggests that we may be on the right track with our fix. However, aside from this mysterious snippet of code in the source, we haven't been able to find any documentation or information on the web indicating that you have to do anything special before finalizing python when using PyWin32. There are also some other calls made there: FreeGatewayModules() and PyEval_RestoreThread(). I'm not sure if we should call these ourselves.

Our fix appears to work, and in the absence of any more information we will probably go with it, but it would be nice to get some confirmation.

Solution

  • Answer came from the python-win32 mailing list: http://mail.python.org/pipermail/python-win32/2013-January/012671.html.

    Unfortunately there are a number of problems repeatedly initializing and finalizing Python, hence the code you saw in pywin32 that no longer attempts to support it. The short-story is simply that doing this is not supported using pywin32 (at least until the issues in Python are fixed via the new module API which supports this, and pywin32 is adjusted accordingly)

    Mark [Hammond]