So, being a Linux guy I stumbled into something pretty puzzling on Windows that I just can't explain.
I have a project structure analougus to this example:
D:\PROJECT
|
| tolkien.py
| __init__.py
|
\---MiddleEarth
| gondor.py
| isengrad.c
| __init__.py
|
\---lib
isengrad.so
Problem: I compile isengrad.c into the shared libary isengrad.so, then load it in gondor.py. My aim is to import gondor.py into tolkien.py.
While gondor.py runs flawlessly when it is run directly, when I import it, the code exits at the point when I load the shared library via ctypes.CDLL, without any error messages.
Reproduction: The content of the files (added some "status messages" to follow where the problem happens):
isengrad.c:
int isengrad(int hobbit){
return hobbit/2;
}
This is then compiled to isengrad.so with
D:\project>chdir MiddleEarth
D:\project\MiddleEarth>gcc -fPIC -shared -o lib/isengrad.so isengrad.c
The shared library is then accessed in gondor.py:
print("started gondor")
import os, ctypes
path_to_isengrad = "D:/project/MiddleEarth/lib/isengrad.so"
print("gondor loads isengrad")
gondor = ctypes.CDLL(path_to_isengrad) # <--- crashes here when imported, not when ran directly
print("gondor loaded isengrad")
gondor.isengrad.argtypes = (ctypes.c_int,)
def faramir(hobbit):
catched_hobbits = gondor.isengrad(hobbit)
return catched_hobbits
if __name__ == '__main__':
print(faramir(5))
print("gondor ran")
print("gondor finished")
which is then imported in tolkien.py:
print("started tolkien")
from MiddleEarth import gondor
print("tolkien imported gondor")
got = gondor.faramir(4)
print(got)
print("tolkien worked")
Now check what happens when I use gondor.py directly VS when I import it in tolkien.py:
D:\project>python MiddleEarth/gondor.py
started gondor
gondor loads isengrad
gondor loaded isengrad
2
gondor ran
gondor finished
D:\project>python tolkien.py
started tolkien
started gondor
gondor loads isengrad
D:\project>
Directly running it causes no problem at all. But importing it causes the whole thing to crash without any word and traceback when loading the shared library. How is this even happening? I even hard-coded the path to the shared library, so different working directory shouldn't be a problem... I didn't have any problem with the very same project on Kubuntu, so this is probably some Windows-related stuff.
Environment:
Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32Windows 10 10.0.17134 Build 17134 (installed on C:)From the moment I saw this question, I wanted to say it's Undefined Behavior (UB). Python comes with its C runtime (UCRTLib), while the Cygwin .dll comes with its own. Mixing compilers and C runtimes in a process, is generally a recipe for disaster.
I found an official statement [Cygwin]: 6.15. Can I link with both MSVCRT*.DLL and cygwin1.dll? (emphasis is mine):
No, you must use one or the other, they are mutually exclusive.
Check [SO]: How to circumvent Windows Universal CRT headers dependency on vcruntime.h (@CristiFati's answer) for more details on MSVCRT*.DLL (VCRuntime*.dll).
Now, the beauty of UB is that it describes a seemingly random behavior.
I've prepared a comprehensive example (slightly modifying your code).
isengrad.c:
#if defined(_WIN32)
# define ISENGRAD_EXPORT_API __declspec(dllexport)
#else
# define ISENGRAD_EXPORT_API
#endif
ISENGRAD_EXPORT_API int isengrad(int hobbit)
{
return hobbit / 2;
}
code00.py:
#!/usr/bin/env python
import ctypes as cts
import sys
DLL_NAME = "./lib/isengrad_{:s}_{:03d}.{:s}".format(
sys.argv[1][:3] if len(sys.argv) > 1 else sys.platform[:3].lower(),
cts.sizeof(cts.c_void_p) * 8,
"dll" if sys.platform[:3].lower() == "win" else "so",
)
print("Attempting to load: {0:s}".format(DLL_NAME))
dll = cts.CDLL(DLL_NAME)
print("DLL Loaded")
def main(*argv):
isengrad = dll.isengrad
isengrad.argtypes = (cts.c_int,)
isengrad.restype = cts.c_int
res = isengrad(46)
print("{:s} returned {:d}".format(isengrad.__name__, res))
if __name__ == "__main__":
print(
"Python {:s} {:03d}bit on {:s}\n".format(
" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32,
sys.platform,
)
)
rc = main(*sys.argv[1:])
print("\nDone.\n")
sys.exit(rc)
code01.py:
#!/usr/bin/env python
import sys
import code00
def main(*argv):
pass
if __name__ == "__main__":
print(
"Python {:s} {:03d}bit on {:s}\n".format(
" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32,
sys.platform,
)
)
rc = main(*sys.argv[1:])
print("\nDone.\n")
sys.exit(rc)
Outputs:
I'll be using 3 windows:
Cmd - Win (032bit and 064bit)
Cygwin's Mintty:
064bit
032bit
Note that even if I'll paste each's contents in one chunk (to avoid scattering them), I switched between them when running the commands
Cygwin 032bit:
[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q056855348]> ~/sopr.sh ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [032bit prompt]> gcc -fPIC -shared -o lib/isengrad_cyg_032.dll isengrad.c [032bit prompt]> ls lib/*.dll lib/isengrad_cyg_032.dll lib/isengrad_win_032.dll lib/isengrad_win_064.dll [032bit prompt]> [032bit prompt]> python ./code00.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin isengrad returned 23 Done. [032bit prompt]> [032bit prompt]> python ./code01.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin Done. [032bit prompt]> [032bit prompt]> python ./code00.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin isengrad returned 23 Done. [032bit prompt]> [032bit prompt]> python3 ./code01.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin Done.
Cygwin 064bit:
[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q056855348]> ~/sopr.sh ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [064bit prompt]> gcc -fPIC -shared -o lib/isengrad_cyg_064.dll isengrad.c [064bit prompt]> ls lib/*.dll lib/isengrad_cyg_032.dll lib/isengrad_cyg_064.dll lib/isengrad_win_032.dll lib/isengrad_win_064.dll [064bit prompt]> [064bit prompt]> python ./code00.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin isengrad returned 23 Done. [064bit prompt]> [064bit prompt]> python ./code01.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin Done. [064bit prompt]> [064bit prompt]> python ./code00.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin isengrad returned 23 Done. [064bit prompt]> [064bit prompt]> python ./code01.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin Done.
Cmd:
[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q056855348]> sopr.bat ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [prompt]> dir /b lib [prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2017\VC\Auxiliary\Build\vcvarsall.bat" x64 ********************************************************************** ** Visual Studio 2017 Developer Command Prompt v15.9.14 ** Copyright (c) 2017 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' [prompt]> cl /nologo /DDLL isengrad.c /link /NOLOGO /DLL /OUT:lib\isengrad_win_064.dll isengrad.c Creating library lib\isengrad_win_064.lib and object lib\isengrad_win_064.exp [prompt]> [prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2017\VC\Auxiliary\Build\vcvarsall.bat" x86 ********************************************************************** ** Visual Studio 2017 Developer Command Prompt v15.9.14 ** Copyright (c) 2017 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x86' [prompt]> cl /nologo /DDLL isengrad.c /link /NOLOGO /DLL /OUT:lib\isengrad_win_032.dll isengrad.c Creating library lib\isengrad_win_032.lib and object lib\isengrad_win_032.exp [prompt]> dir /b lib\*.dll isengrad_cyg_032.dll isengrad_cyg_064.dll isengrad_win_032.dll isengrad_win_064.dll [prompt]> set _PATH=%PATH% [prompt]> :: ----- Python 032bit ----- [prompt]> set PATH=%_PATH%;e:\Install\x86\Cygwin\Cygwin\Version\bin [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" ./code00.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" ./code01.py win Attempting to load: ./lib/isengrad_win_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 Done. [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" ./code00.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" ./code01.py cyg Attempting to load: ./lib/isengrad_cyg_032.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32 Done. [prompt]> :: ----- Python 064bit ----- [prompt]> set PATH=%_PATH%;c:\Install\x64\Cygwin\Cygwin\AllVers\bin [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" ./code00.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" ./code01.py win Attempting to load: ./lib/isengrad_win_064.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32 Done. [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" ./code00.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll DLL Loaded Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32 isengrad returned 23 Done. [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" ./code01.py cyg Attempting to load: ./lib/isengrad_cyg_064.dll [prompt]> [prompt]> echo %errorlevel% -1073741819
As seen, cross compiler .exe - .dll worked in 7 (out of 8) cases (crashed on 064bit Win Python with code01.py), while the same compiler worked in all 8 of them.
So, I'd advise that when playing with such environments, try to keep the compilers used to build various parts consistent (or compatible at least).
Similar question:
I just thought of a reason why things could go wrong on 064bit: sizeof(long) generally differs (sizes below are in bytes):
4 on Win
8 on Cygwin (on Nix, in general)
Same thing for sizeof(long double) (which is generally 2 * sizeof(long)).
Check:
[SO]: Maximum and minimum value of C types integers from Python (@CristiFati's answer)
[SO]: _csv.Error: field larger than field limit (131072) (@CristiFati's answer)
So, if the Cygwin .dll exposes some long value greater than 2 ** 64 (1 << 64), it will be truncated in the Win process, and in this case a crash might occur. Theoretically, this situation should affect the reverse scenario as well, but it doesn't.
There are other factors that might lead to this behavior, like default memory alignment, and so on.