Cannot find internal module with imp

Question

While trying to run the algolia fork of the mongo_connector I have encounted a problem selecting the algolia doc manager as an internal module. It seems to me that everybody else relies on adding the doc manager as an external source although really shouldn't be necessary.

To install I have created this requirement.txt:

algoliasearch==1.5.0
elasticsearch==0.4.5
pymongo==2.8
pysolr==3.3.0
requests==2.5.1
urllib3==1.10
wsgiref==0.1.2
git+https://github.com/algolia/mongo-connector.git

And installed it with sudo pip install -r requirements.txt.

Then running it:

mongo-connector -m mongodb://localhost:5001/meteor -n meteor.items -t credentials-etc:items -o items.txt -d doc_managers/algolia_doc_manager

Gives me:

 Traceback (most recent call last):
   File "/usr/local/bin/mongo-connector", line 9, in <module>
     load_entry_point('mongo-connector==1.3.dev0', 'console_scripts', 'mongo-connector')()
   File "build/bdist.linux-x86_64/egg/mongo_connector/connector.py", line 720, in main
   File "build/bdist.linux-x86_64/egg/mongo_connector/connector.py", line 77, in __init__
   File "build/bdist.linux-x86_64/egg/mongo_connector/connector.py", line 64, in load_doc_manager
 IOError: [Errno 2] No such file or directory

Trying to fix it, I looked up the source at which it fails:

def load_doc_manager(path):
    name, _ = os.path.splitext(os.path.basename(path))
    try:
        from importlib.machinery import SourceFileLoader
        loader = SourceFileLoader(name, path)
        module = loader.load_module(name)
    except ImportError:
        module = imp.load_source(name, path)
    return module

So with path being doc_managers/algolia_doc_manager why is imp.load_source() not able to find it when the file is there? Could it be because the package is installed as .egg file? Also tried running with other doc-managers, with same results.

Only when I provide an absolute URL to the doc-manager, it works.

Answer 1

I ended up resolving doc manager path programmatically like so

import mongo_connector.doc_managers.algolia_doc_manager as doc_manager
import os.path as path, sys

sys.stdout.write(path.join(path.dirname(doc_manager.__file__),'algolia_doc_manager.py'))