Cannot find module after change directory

So I have to create a run folder, that is, a folder which has a bunch of python files which I need to run. I am able to create this folder with ease and all the files are there. However, when I try to run the files using importlib, python is not about to find it.

I want to make sure that the code running is, in fact, the run folder code, hence I change the directory to location of the run folder.

abs_path = os.path.abspath("{}".format(run_location))
os.chdir(abs_path)
files = [f for f in os.listdir('.') if os.path.isfile(f)]
print(files)
try:
    driver_module = importlib.import_module("main_driver.py")
    driver_module.main(config, logger)
except Exception as e:
    logger.error("error", str(e))
    finish_fail(config, logger)
finish_success(config, logger)

In the above example, I just want to run main_driver.py. The output from the above is:

['PrepareDataframe.py', 'categorical_encoding.py', 'extra_files.zip', 'build_features.py', 'spot_extractor.py', 'dev.ini', 'featuriser.py', 'main_driver.py', 'time_features_extract.py']

Clearly, main_driver.py is within the current working directory but I get this error.

No module named 'main_driver'

Traceback:

Traceback (most recent call last):
  File "./utils/submit.py", line 292, in <module>
    driver_module = importlib.import_module("main_driver")
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'main_driver'

Solution

This appears to be a PYTHONPATH issue.

When you run the Python interactive interpreter, the first item in sys.path is an empty string, which effectively means "the current directory". So any time you try to import a module, it will look in the current directory first.

However, if you run python myscript.py, this is not the case -- the first item in sys.path is the actual directory you were in when you ran the script. So, if you use os.chdir() to change the current directory, the new current directory isn't in sys.path, and so import_module('foo') fails because it's not found.

If you want to mimic the interactive python behavior of always looking in the current directory for modules, add sys.path.insert(0, '') to your script.