MinGW / SWIG built python extension failing to load in Anaconda command terminal

Question

I have been experimenting with using SWIG to wrap C code for use in Python.I am using MSYS and MingGW on a Windows 10 platform. I have written a small test library

#include "add.h"

double add(double a, double b)
{
  double c = a+b;
  return c;
}

with header

#ifndef __ADD_H
#define __ADD_H

extern double add(double a, double b);

#endif

The interface file is testpackage.i

%module testpackage
%{
#include "add.h"
%}

double add(double, double);

This is then compiled

swig -python -py3 testpackage.i
gcc -c -Wall add.c add_wrap.c
gcc -shared add.o add_wrap.o -L/mingw64/lib -lpython3.9 -o _testpackage.pyd

A small python program to test this:

import testpackage

x = 2.3
y = 5.6

z = testpackage.add(x,y)

print(z)

Now this compiles without errors or warnings and runs perfectly in the MSYS terminal. (I also wrote some more complex multi-file libraries which also worked fine) The python program resides in the same directory as the generated testpackage.py and _testpackage.pyd files. However, if I try to run the python script in my Anaconda cmd or PowerShell terminals, or if I load it into Spyder and run from there. I get the error:

(base) C:\msys64\home\hoyla\swigHelloWorld>python test1.py
Traceback (most recent call last):
  File "test1.py", line 1, in <module>
    import testpackage
  File "C:\msys64\home\hoyla\swigHelloWorld\testpackage.py", line 15, in <module>
    import _testpackage
ImportError: DLL load failed while importing _testpackage: The specified module could not be found.

Now I know that the Python which ships with MinGW and is running in the MSYS terminal is 3.9.6 whereas the Anaconda Python is 3.8.11. Could this be the issue? If this is the case, how can I build my library with backwards compatibility in mind? Or am I just missing something else here?

Update: Managed to install 3.9.6 in a conda environment and still get module could not be found - so that's not the problem!

Further update: Running the Dependencies (x64) I discovered that _testpackage.pyd requires libPython3.9.dll (obvs - since I linked it!). However, outside of the MSYS terminal this can't be found as the folder was not in the PATH. I added it to the PATH, dependency walker now locates it and the error message is gone when I run from Anaconda console. However now the test script crashes with no error message at all if I import my testpackage module - either running my test py script or if I import in the interactive python - python just exits. Everything still works find in MSYS console.

Answer 1

Well after much anguish I worked it out. Python.exe under MSYS/MinGW is compiled via GCC. See the prompt when starting python from the MSYS command line:

$ python
Python 3.9.6 (default, Aug 15 2021, 14:43:38)  [GCC UCRT 10.3.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Whereas the python from Anaconda is compiled Microsoft C

python
Python 3.9.6 (default, Aug 18 2021, 15:44:49) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

As a result the two python exe's are linked to different C runtime libraries. My module / DLL compiled under GCC is linked to the same one as the MSYS python so runs fine. But when run by Anaconda's python causes some deep issues with the runtime library.

The work around is to compile (in GCC under MSYS) linking to Anaconda's python library rather than MSYS's so the link step from the original question becomes:

gcc -shared add.o add_wrap.o -L/c/Users/$(USERNAME)/Anaconda3/envs/$(ENVIRONMENT)/libs -lpython39 -o _testpackage.pyd

Generally replace $(USERNAME) with the WIndows username and $(ENVIRONMENT) with the Anaconda environment you're targetting. The L path will probably differ but basically it is the libs directory for the Anaconda environment. Note the lib in anaconda is python39 where as the one in MingGW is called python3.9. I think the Anaconda one is linking statically. But this now works, however I can only build for either Anaconda python or Mingw python not both even though they are essentially the same version. Which seems kind of a pain - particularly if I was planning on distributing the module - which luckily I'm not!