As far as I understand, one of the main advantages of distributing Python packages through wheels is that I can include extension modules in a compiled form. Then, the user of the package is not required to have a system that allows compilation of the source code.
Now I managed to build a wheel for my package that includes a Fortran extension module. The computer on which I built has Windows7 64, and Python 3.6.
In order to get everything running, I followed this very helpful guideline (many thanks to Michael Hirsch). One of the steps was to install MinGW-64 with the following settings: Architecture: x86_64, Threads: posix, Exception: seh.
I then installed the Python package on another testing machine (Win10 64, Python 3.6) from that wheel:
D:\dist2>pip install SMUTHI-0.2.0a0-cp36-cp36m-win_amd64.whl
Processing d:\dist2\smuthi-0.2.0a0-cp36-cp36m-win_amd64.whl
Requirement already satisfied: scipy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: sympy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: argparse in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: numpy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: matplotlib in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: pyyaml in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: six>=1.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: python-dateutil in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pytz in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: cycler>=0.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Installing collected packages: SMUTHI
Successfully installed SMUTHI-0.2.0a0
However, when I started a test run of the program, I encountered the following error:
D:\dist2>smuthi example_input.dat
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\Scripts\smuthi.exe\__main__.py", line 5, in <module>
File "c:\programdata\anaconda3\lib\site-packages\smuthi\__main__.py", line 4, in <module>
import smuthi.read_input
File "c:\programdata\anaconda3\lib\site-packages\smuthi\read_input.py", line 3, in <module>
import smuthi.simulation
File "c:\programdata\anaconda3\lib\site-packages\smuthi\simulation.py", line 8, in <module>
import smuthi.t_matrix as tmt
File "c:\programdata\anaconda3\lib\site-packages\smuthi\t_matrix.py", line 6, in <module>
import smuthi.nfmds.t_matrix_axsym as nftaxs
File "c:\programdata\anaconda3\lib\site-packages\smuthi\nfmds\t_matrix_axsym.py", line 11, in <module>
import smuthi.nfmds.taxsym
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.
The extension .pyd file (taxsym.cp36-win_amd64.pyd) was at its place - just Python couldn't load it.
Next, I uninstalled MinGW from the testing machine and reinstalled MinGW-64 with the same settings that I had used on the building machine (see above). Afterwards, I could run the program, and Python was able to correctly load the extension module.
My question is: Does anybody have an idea why the error occurred in the first place? And how can I avoid that the user of my Python package has to have a specific version of MinGW installed (or even any) for the package to work properly?
Edit: A small example that reproduces the error:
File structure:
setup.py
example/
__init__.py
run_hello.py
extension_package/
__init__.py
fortran_hello.f90
The setup.py reads:
import setuptools
from numpy.distutils.core import Extension
from numpy.distutils.core import setup
setup(
name="example",
version="0.1",
author="My Name",
author_email="[email protected]",
description="Example package to demonstrate wheel issue",
packages=['example', 'example.extension_package'],
ext_modules=[Extension('example.extension_package.fortran_hello',
['example/extension_package/fortran_hello.f90'])],
)
The run_hello.py reads:
import example.extension_package.fortran_hello
example.extension_package.fortran_hello.hello()
The fortran_hello.f90 reads:
subroutine hello
print *,"Hello World!"
end subroutine hello
I ran python setup.py bdist_wheel which resulted in the file example-0.1-cp36-cp36m-win_amd64.whl
D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1
D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
Hello World!
>>> exit()
This is as it should be.
To reproduce the error, I renamed the MinGW folder on the testing machine to some other name and then:
D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1
D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\site-packages\example\run_hello.py", line 1, in <module>
import example.extension_package.fortran_hello
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.
I recently ran into this issue by writing my own f2py building tool chain by compiling and linking all the components individually. The script was finding or installing required compilers automatically if they werent already found on the path. For cases where the gfortran tools werent on the path, but were present on the machine, I was able to inject the correct environment variables to os.environ and spawn compiler calls using Popen and the set of environment variables so that the pyd would compile. But outside of that python instance the environment variables were not correct for the pyd to run, I was getting the same DLL load failed error even on the same computer that compiled the pyds but which didnt have the correct paths setup.
So, since I'm compiling all steps separately, only using f2py to generate the f and c wrappers, I simply added -static -static-libgfortran -static-libgcc to my link step, and this causes the pyd to include the required libraries to run on those machines without the correct environment variables.
Achieving the same using numpy.distutils is possible (thanks to https://github.com/numpy/numpy/issues/3405):
from numpy.distutils.core import Extension, setup
if __name__ == "__main__":
setup(
name="this",
ext_modules=[
Extension("fortmod_nostatic",
["src/code.f90"],
),
Extension("fortmod_withstatic",
["src/code.f90"],
extra_link_args=["-static", "-static-libgfortran", "-static-libgcc"]
)
]
)
I put the above in a file test.py and built with python test.py build_ext --inplace --compiler=mingw32 --fcompiler=gnu95 -f
For comparison there is a clear size difference. Inspecting the pyd's with dependency walker shows the nostatic one depends on libgfortran-4.dll whereas the extra flags generate a pyd that does not depend on this library. In my case after adding the static flags the machine without correct environment variables is able to run the pyds, and I suspect this case will be similar to yours since the dependency on libgfortran is removed.
Hope that helps! my first SO post..
