Search code examples
pythonpython-2.7anacondaintel-mklminiconda

python "Intel MKL FATAL ERROR: Cannot load libmkl_rt.so"

I am running multiple python ML jobs at same time. Most of the time my python script runs fine but when there are too many jobs starting at the same time, it fails with this error.

It is also hard to reproduce this error as I am unable to find the cause of it. The file is present at the location.

Error:

Intel MKL FATAL ERROR: Cannot load /usr/local/miniconda-2.7.13/lib/python2.7/site-packages/numpy/core/../../../../libmkl_rt.so.
Solution

  • First I created a new conda virtual enviornment to use with multithreading:

    conda create -n nomkl_env python=3.7 numpy nomkl

    I activated this environment and ran this command via the conda MKL docs:

    conda install numpy scipy scikit-learn numexpr

    Now I am able to run multiple instances of Python at the same time. Earlier when I ran conda remove mkl mkl-service it somehow removed python from the bin of an existing virutal env. This was sub-optimal. I would up creating a dedicated conda env and this did the trick.