Search code examples
pythonmacoslapackopenblas

'lapacke.h' file not found error when installing libact


I am trying to install the python libact package on my computer using the tutorial on their github page. I'm using python 3.7.5 on macOS.

I have installed openblas as a requirement which consists of lapacke.h header file. However, when I type any of the commands for installing libact, I get the error libact/query_strategies/src/variance_reduction/variance_reduction.c:5:10: fatal error: 'lapacke.h' file not found. I searched this issue on github and related questions on SO and none of these solutions have worked for me.

I have tried adding the lapacke.h directory to my path environmental variable as suggested here, reinstalling gcc, and installing the package using other methods but none of them have worked.

Any help would be appreciated.

Here is the full message when I try to install the package:

Collecting libact
  Using cached https://files.pythonhosted.org/packages/4b/8c/1f57cbe9e6b647afa14f8446c20caa2343fa26153be179e5a89060cd6ec2/libact-0.1.6.tar.gz
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from libact) (41.0.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/site-packages (from libact) (1.17.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/site-packages (from libact) (1.3.1)
Requirement already satisfied: scikit-learn<=0.19.2 in /usr/local/lib/python3.7/site-packages (from libact) (0.19.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/site-packages (from libact) (3.1.1)
Requirement already satisfied: Cython in /usr/local/lib/python3.7/site-packages (from libact) (0.29.14)
Requirement already satisfied: joblib in /usr/local/lib/python3.7/site-packages (from libact) (0.13.2)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/site-packages (from matplotlib->libact) (2.8.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/site-packages (from matplotlib->libact) (1.1.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/site-packages (from matplotlib->libact) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/site-packages (from matplotlib->libact) (2.4.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil>=2.1->matplotlib->libact) (1.12.0)
Building wheels for collected packages: libact
  Building wheel for libact (setup.py) ... error
  ERROR: Complete output from command /usr/local/opt/python/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/jl/ldl4kwz12vgdf44bt2nc70zw0000gn/T/pip-install-6402bf85/libact/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/jl/ldl4kwz12vgdf44bt2nc70zw0000gn/T/pip-wheel-qhxixz3i --python-tag cp37:
  ERROR: Platform Detection: Mac OS X. Link to openblas...
  Build VarianceReduction...
  Build HintSVM...
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14-x86_64-3.7
  creating build/lib.macosx-10.14-x86_64-3.7/libact
  copying libact/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact
  creating build/lib.macosx-10.14-x86_64-3.7/libact/base
  copying libact/base/interfaces.py -> build/lib.macosx-10.14-x86_64-3.7/libact/base
  copying libact/base/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/base
  copying libact/base/dataset.py -> build/lib.macosx-10.14-x86_64-3.7/libact/base
  creating build/lib.macosx-10.14-x86_64-3.7/libact/models
  copying libact/models/perceptron.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
  copying libact/models/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
  copying libact/models/sklearn_adapter.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
  copying libact/models/logistic_regression.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
  copying libact/models/svm.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
  creating build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
  copying libact/models/multilabel/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
  copying libact/models/multilabel/binary_relevance.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
  copying libact/models/multilabel/dummy_clf.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
  creating build/lib.macosx-10.14-x86_64-3.7/libact/labelers
  copying libact/labelers/ideal_labeler.py -> build/lib.macosx-10.14-x86_64-3.7/libact/labelers
  copying libact/labelers/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/labelers
  copying libact/labelers/interactive_labeler.py -> build/lib.macosx-10.14-x86_64-3.7/libact/labelers
  creating build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/random_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/hintsvm.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/quire.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/active_learning_by_learning.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/variance_reduction.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/query_by_committee.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/density_weighted_meta.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/density_weighted_uncertainty_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  copying libact/query_strategies/uncertainty_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
  creating build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
  copying libact/query_strategies/multilabel/cost_sensitive_reference_pair_encoding.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
  copying libact/query_strategies/multilabel/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
  copying libact/query_strategies/multilabel/maximum_margin_reduction.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
  copying libact/query_strategies/multilabel/binary_minimization.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
  copying libact/query_strategies/multilabel/multilabel_with_auxiliary_learner.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
  copying libact/query_strategies/multilabel/adaptive_active_learning.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
  creating build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
  copying libact/query_strategies/multiclass/expected_error_reduction.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
  copying libact/query_strategies/multiclass/mdsp.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
  copying libact/query_strategies/multiclass/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
  copying libact/query_strategies/multiclass/active_learning_with_cost_embedding.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
  copying libact/query_strategies/multiclass/hierarchical_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
  creating build/lib.macosx-10.14-x86_64-3.7/libact/utils
  copying libact/utils/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/utils
  running build_ext
  building 'libact.query_strategies._variance_reduction' extension
  Warning: Can't read registry to find the necessary compiler setting
  Make sure that Python modules winreg, win32api or win32con are installed.
  C compiler: clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers

  creating build/temp.macosx-10.14-x86_64-3.7
  creating build/temp.macosx-10.14-x86_64-3.7/libact
  creating build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies
  creating build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies/src
  creating build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies/src/variance_reduction
  compile options: '-I/usr/local/lib/python3.7/site-packages/numpy/core/include -I/opt/local/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c'
  extra options: '-std=c11'
  clang: libact/query_strategies/src/variance_reduction/variance_reduction.c
  libact/query_strategies/src/variance_reduction/variance_reduction.c:5:10: fatal error: 'lapacke.h' file not found
  #include <lapacke.h>
           ^~~~~~~~~~~
  1 error generated.
  error: Command "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/lib/python3.7/site-packages/numpy/core/include -I/opt/local/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c libact/query_strategies/src/variance_reduction/variance_reduction.c -o build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies/src/variance_reduction/variance_reduction.o -std=c11" failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for libact
  Running setup.py clean for libact
Failed to build libact
Installing collected packages: libact
  Running setup.py install for libact ... error
    ERROR: Complete output from command /usr/local/opt/python/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/jl/ldl4kwz12vgdf44bt2nc70zw0000gn/T/pip-install-6402bf85/libact/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/jl/ldl4kwz12vgdf44bt2nc70zw0000gn/T/pip-record-26z7vy46/install-record.txt --single-version-externally-managed --compile:
    ERROR: Platform Detection: Mac OS X. Link to openblas...
    Build VarianceReduction...
    Build HintSVM...
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14-x86_64-3.7
    creating build/lib.macosx-10.14-x86_64-3.7/libact
    copying libact/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact
    creating build/lib.macosx-10.14-x86_64-3.7/libact/base
    copying libact/base/interfaces.py -> build/lib.macosx-10.14-x86_64-3.7/libact/base
    copying libact/base/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/base
    copying libact/base/dataset.py -> build/lib.macosx-10.14-x86_64-3.7/libact/base
    creating build/lib.macosx-10.14-x86_64-3.7/libact/models
    copying libact/models/perceptron.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
    copying libact/models/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
    copying libact/models/sklearn_adapter.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
    copying libact/models/logistic_regression.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
    copying libact/models/svm.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models
    creating build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
    copying libact/models/multilabel/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
    copying libact/models/multilabel/binary_relevance.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
    copying libact/models/multilabel/dummy_clf.py -> build/lib.macosx-10.14-x86_64-3.7/libact/models/multilabel
    creating build/lib.macosx-10.14-x86_64-3.7/libact/labelers
    copying libact/labelers/ideal_labeler.py -> build/lib.macosx-10.14-x86_64-3.7/libact/labelers
    copying libact/labelers/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/labelers
    copying libact/labelers/interactive_labeler.py -> build/lib.macosx-10.14-x86_64-3.7/libact/labelers
    creating build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/random_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/hintsvm.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/quire.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/active_learning_by_learning.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/variance_reduction.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/query_by_committee.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/density_weighted_meta.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/density_weighted_uncertainty_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    copying libact/query_strategies/uncertainty_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies
    creating build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
    copying libact/query_strategies/multilabel/cost_sensitive_reference_pair_encoding.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
    copying libact/query_strategies/multilabel/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
    copying libact/query_strategies/multilabel/maximum_margin_reduction.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
    copying libact/query_strategies/multilabel/binary_minimization.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
    copying libact/query_strategies/multilabel/multilabel_with_auxiliary_learner.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
    copying libact/query_strategies/multilabel/adaptive_active_learning.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multilabel
    creating build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
    copying libact/query_strategies/multiclass/expected_error_reduction.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
    copying libact/query_strategies/multiclass/mdsp.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
    copying libact/query_strategies/multiclass/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
    copying libact/query_strategies/multiclass/active_learning_with_cost_embedding.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
    copying libact/query_strategies/multiclass/hierarchical_sampling.py -> build/lib.macosx-10.14-x86_64-3.7/libact/query_strategies/multiclass
    creating build/lib.macosx-10.14-x86_64-3.7/libact/utils
    copying libact/utils/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/libact/utils
    running build_ext
    building 'libact.query_strategies._variance_reduction' extension
    Warning: Can't read registry to find the necessary compiler setting
    Make sure that Python modules winreg, win32api or win32con are installed.
    C compiler: clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers

    creating build/temp.macosx-10.14-x86_64-3.7
    creating build/temp.macosx-10.14-x86_64-3.7/libact
    creating build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies
    creating build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies/src
    creating build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies/src/variance_reduction
    compile options: '-I/usr/local/lib/python3.7/site-packages/numpy/core/include -I/opt/local/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c'
    extra options: '-std=c11'
    clang: libact/query_strategies/src/variance_reduction/variance_reduction.c
    libact/query_strategies/src/variance_reduction/variance_reduction.c:5:10: fatal error: 'lapacke.h' file not found
    #include <lapacke.h>
             ^~~~~~~~~~~
    1 error generated.
    error: Command "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/lib/python3.7/site-packages/numpy/core/include -I/opt/local/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c libact/query_strategies/src/variance_reduction/variance_reduction.c -o build/temp.macosx-10.14-x86_64-3.7/libact/query_strategies/src/variance_reduction/variance_reduction.o -std=c11" failed with exit status 1
    ----------------------------------------
ERROR: Command "/usr/local/opt/python/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/jl/ldl4kwz12vgdf44bt2nc70zw0000gn/T/pip-install-6402bf85/libact/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/jl/ldl4kwz12vgdf44bt2nc70zw0000gn/T/pip-record-26z7vy46/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/jl/ldl4kwz12vgdf44bt2nc70zw0000gn/T/pip-install-6402bf85/libact/

Solution

    1. install openblas and lapack:
    brew install openblas    
    brew install lapack
    
    1. Then add them in your ~/.bashrc or ~/.zshrc or just execute these commands in your terminal session:
    export LDFLAGS="-L/usr/local/opt/lapack/lib"
    export CPPFLAGS="-I/usr/local/opt/lapack/include"
    export PKG_CONFIG_PATH="/usr/local/opt/lapack/lib/pkgconfig"
    
    export LDFLAGS="-L/usr/local/opt/openblas/lib"
    export CPPFLAGS="-I/usr/local/opt/openblas/include"
    export PKG_CONFIG_PATH="/usr/local/opt/openblas/lib/pkgconfig"
    

    Now, there is just one problem left:
    openblas and lapack has a function with the same name but different signatures and hence you will get this error if you run pip install libact:

    libact/query_strategies/src/variance_reduction/variance_reduction.c:8:13: error: conflicting types for 'dgesvd_'
        extern void LAPACK_dgesvd( char* jobu, char* jobvt, int* m, int* n, double* a,...
    
    1. One fix is to let go of the variance reduction algorithm in libact for now! The rest will work just fine:
    LIBACT_BUILD_HINTSVM=1  LIBACT_BUILD_VARIANCE_REDUCTION=0 pip install git+https://github.com/ntucllab/libact.git
    

    This will install libact without error.