relative imports on CDSW

Question

I have a project on CDSW organized as follow :

/home/cdsw/my_project_v2.1
   |_>input
   |_>output
   |_>scr
      |_>__init__.py
      |_>main.py
      |_>utils
          |_>__init__.py
          |_>helpers.py

in my current code, I use sys.path.append to perform my imports.

import sys

sys.path.append("/home/cdsw/my_project_v2.1/src/utils/")

from helpers import bar

This works fine but it is a bad practice because if the version change, then I need to change all my scripts that use the path.

I wanted to replace it with some relative path :

from .utils.helpers import bar

But I got the error :

$ pwd
/home/cdsw
$ python3 my_project_v2.1/src/main.py 
Traceback (most recent call last):
  File "my_project_v2.1/src/main.py", line 1, in <module>
    from .utils.helpers import bar
ModuleNotFoundError: No module named '__main__.helpers'; '__main__' is not a package

what do I need to change in my architecture or in my code to make it work ?

Answer 1

Just use

from utils.helpers import bar

A short excerpt from the documentation of the Python command line arguments:

If the script name refers directly to a Python file, the directory containing that file is added to the start of sys.path, and the file is executed as the __main__ module.

The first half of the sentence means that you can use absolute module names when referring to the contents of your directory, because Python will search for module there. The fact that you can not use relative imports is a consequence of the second half of the sentence.

As a side note, you may also consider omitting the version number from the name of the directory, or better yet, put your code directly in /home/cdsw. The latter may sound strange as you would never do that on a regular machine, but here everything is in a container and actually this is the way your code is supposed to be organized in CDSW. You can confirm this by creating a new project based on a template or a git URL – both will put code directly in the home directory.