Python 3 script using relative imports on standard input gives error: No module named '__main__.XXX'; '__main__' is not a package

Question

Python 3 script using relative imports on standard input gives error: No module named 'main.XXX'; 'main' is not a package

If I have a Python 3 script that uses a relative import statement of the form

from .subscript2 import subfunc2

And some other script imports the above script as a module, it works without issue. However, when I execute the script containing the above line on standard input to the Python interpreter, I get an error of the form:

ModuleNotFoundError: No module named '__main__.subscript2'; '__main__' is not a package

The reason I need to use stdin, is because I need to use Elpy (https://elpy.readthedocs.io/en/latest/) that allows me to evaluate that script (using the C-c C-c key binding) and then add additional Python code to call functions in that script, akin to the pdb 'interact' command.

Below is the self-contained script to demonstrate what is happening:

#!/bin/bash

rm -rf /tmp/topdir
mkdir /tmp/topdir
mkdir /tmp/topdir/subdir

unset PYTHONPATH

cat > /tmp/topdir/topscript.py <<'EOF'
from subdir.subscript1 import subfunc1
EOF

touch /tmp/topdir/subdir/__init__.py

cat > /tmp/topdir/subdir/subscript1.py <<'EOF'
from .subscript2 import subfunc2

def subfunc1():
    subfunc2()

print('subscript1 loaded successfully')
EOF

cat > /tmp/topdir/subdir/subscript2.py <<'EOF'
import os
def subfunc2():
    print('subfunc2 called. Current working directory: {}'.format(os.getcwd()))

print('subscript2 loaded successfully')
subfunc2()
EOF

echo
echo ////////////////////////////////////////////////////////////////////////////////
echo Python version:
echo ////////////////////////////////////////////////////////////////////////////////
python --version

cd /tmp
echo
echo ////////////////////////////////////////////////////////////////////////////////
echo Direct execution of topscript.py from inside $(pwd)
echo ////////////////////////////////////////////////////////////////////////////////
python /tmp/topdir/topscript.py

cd /tmp/topdir
echo
echo ////////////////////////////////////////////////////////////////////////////////
echo Using stdin on topscript.py from inside $(pwd):
echo ////////////////////////////////////////////////////////////////////////////////
python < topscript.py

cd /tmp/topdir/subdir
echo
echo ////////////////////////////////////////////////////////////////////////////////
echo Using stdin on subscript1.py from inside $(pwd):
echo ////////////////////////////////////////////////////////////////////////////////
python < subscript1.py

cd /tmp/topdir/subdir
echo
echo ////////////////////////////////////////////////////////////////////////////////
echo Using -m subscript1 from inside $(pwd):
echo ////////////////////////////////////////////////////////////////////////////////
python -m subscript1

cd /tmp/topdir/subdir
echo
echo ////////////////////////////////////////////////////////////////////////////////
echo Using stdin on subscript2.py from inside $(pwd):
echo ////////////////////////////////////////////////////////////////////////////////
python < subscript2.py

# This one was added in response to https://stackoverflow.com/a/55895684/257924
cd /tmp/topdir
echo
echo ////////////////////////////////////////////////////////////////////////////////
echo Using -m on subdir.subscript1 from inside $(pwd):
echo ////////////////////////////////////////////////////////////////////////////////
python -m subdir.subscript1

# This one is a variation of https://stackoverflow.com/a/55895684/257924 using symlinks:
cd /tmp/topdir/subdir
echo
echo ////////////////////////////////////////////////////////////////////////////////
echo "Using -m on selfdir.subscript1 from inside $(pwd) (where selfdir is a symlink):"
echo ////////////////////////////////////////////////////////////////////////////////
ln -s ../subdir selfdir
python -m selfdir.subscript1

Here is the output of the above:

////////////////////////////////////////////////////////////////////////////////
Python version:
////////////////////////////////////////////////////////////////////////////////
Python 3.7.3

////////////////////////////////////////////////////////////////////////////////
Direct execution of topscript.py from inside /tmp
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp
subscript1 loaded successfully

////////////////////////////////////////////////////////////////////////////////
Using stdin on topscript.py from inside /tmp/topdir:
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir
subscript1 loaded successfully

////////////////////////////////////////////////////////////////////////////////
Using stdin on subscript1.py from inside /tmp/topdir/subdir:
////////////////////////////////////////////////////////////////////////////////
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named '__main__.subscript2'; '__main__' is not a package

////////////////////////////////////////////////////////////////////////////////
Using -m subscript1 from inside /tmp/topdir/subdir:
////////////////////////////////////////////////////////////////////////////////
Traceback (most recent call last):
  File "/home/drunkard/conda/Ubuntu.18.04.miniconda3/envs/envpython3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/drunkard/conda/Ubuntu.18.04.miniconda3/envs/envpython3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/topdir/subdir/subscript1.py", line 1, in <module>
    from .subscript2 import subfunc2
ImportError: attempted relative import with no known parent package

////////////////////////////////////////////////////////////////////////////////
Using stdin on subscript2.py from inside /tmp/topdir/subdir:
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir/subdir

////////////////////////////////////////////////////////////////////////////////
Using -m on subdir.subscript1 from inside /tmp/topdir:
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir
subscript1 loaded successfully

////////////////////////////////////////////////////////////////////////////////
Using -m on selfdir.subscript1 from inside /tmp/topdir/subdir (where selfdir is a symlink):
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir/subdir
subscript1 loaded successfully

I've tried various things and could not come up with a solution. The best I could do is delete the initial "." like this:

from subscript2 import subfunc2

but it breaks normal execution:

////////////////////////////////////////////////////////////////////////////////
Python version:
////////////////////////////////////////////////////////////////////////////////
Python 3.7.3

////////////////////////////////////////////////////////////////////////////////
Direct execution of topscript.py from inside /tmp
////////////////////////////////////////////////////////////////////////////////
Traceback (most recent call last):
  File "/tmp/topdir/topscript.py", line 1, in <module>
    from subdir.subscript1 import subfunc1
  File "/tmp/topdir/subdir/subscript1.py", line 1, in <module>
    from subscript2 import subfunc2
ModuleNotFoundError: No module named 'subscript2'

////////////////////////////////////////////////////////////////////////////////
Using stdin on topscript.py from inside /tmp/topdir:
////////////////////////////////////////////////////////////////////////////////
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/topdir/subdir/subscript1.py", line 1, in <module>
    from subscript2 import subfunc2
ModuleNotFoundError: No module named 'subscript2'

////////////////////////////////////////////////////////////////////////////////
Using stdin on subscript1.py from inside /tmp/topdir/subdir:
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir/subdir
subscript1 loaded successfully

////////////////////////////////////////////////////////////////////////////////
Using -m subscript1 from inside /tmp/topdir/subdir:
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir/subdir
subscript1 loaded successfully

////////////////////////////////////////////////////////////////////////////////
Using stdin on subscript2.py from inside /tmp/topdir/subdir:
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir/subdir

////////////////////////////////////////////////////////////////////////////////
Using -m on subdir.subscript1 from inside /tmp/topdir:
////////////////////////////////////////////////////////////////////////////////
Traceback (most recent call last):
  File "/home/drunkard/conda/Ubuntu.18.04.miniconda3/envs/envpython3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/drunkard/conda/Ubuntu.18.04.miniconda3/envs/envpython3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/topdir/subdir/subscript1.py", line 1, in <module>
    from subscript2 import subfunc2
ModuleNotFoundError: No module named 'subscript2'

////////////////////////////////////////////////////////////////////////////////
Using -m on selfdir.subscript1 from inside /tmp/topdir/subdir (where selfdir is a symlink):
////////////////////////////////////////////////////////////////////////////////
subscript2 loaded successfully
subfunc2 called. Current working directory: /tmp/topdir/subdir
subscript1 loaded successfully

In order to make the above work, I would have to set PYTHONPATH in the environment to the colon-separated list of the directories with all of the scripts I use. That would work, for my own scripts, but just as soon as I want to read other published scripts into python via stdin, it will break again unless I go through those scripts and delete the initial "." temporarily (which is a hassle and error-prone).

So, is there a way to make this work for all of those cases? I do have the option to change the tool I'm using (the one that feeds the script to python's stdin) to add additional code to that stdin stream, before the actual script, but if that is the solution, what statements would force it to work?

Helpful reference material to aid in understanding, but that did not reveal the solution:

• intra-package-references
• https://stackoverflow.com/a/8195271/257924
• The Imports section of https://hackaday.com/2018/08/15/stop-using-python-2-what-you-need-to-know-about-python-3/
• https://realpython.com/absolute-vs-relative-python-imports/
• https://stackoverflow.com/a/14132912/257924

Answer 1

I spent a lot of time looking at clever alternatives to allow me to do what I want, and all of the solutions I could think of were just too complicated to put into practice, so:

I ended up avoiding this by using symbolic links in the same directory in which the top-level script resides. It's not great, but its better than having to change PYTHONPATH in the working environment. So if I have utility modules stored in my ~/bin/python directory, but I'm actually working on a script in, say, ~/some_project_dir/my_top_level.py, I create a symbolic link like this:

ln -s ~/bin/python ~/some_project_dir/python

then inside the my_top_level.py file, I use import statements such as:

from python.file_utils import read_lines_from_file
lines = read_lines_from_file("/tmp/some_file")

So if I ever relocate ~/some_project_dir to some other directory, the symbolic link goes with it. If I ever were to move the ~/bin/python directory, then I do have the chore of updating the symbolic links, but that is a minor chore, and doesn't happen all that much.