Search code examples
pythonpackagerelative-path

python packaging not running, treated like a module

I am trying to create a package in python 3.11.6 on Windows 10 which has relative paths to access the various code modules, Firstly, I thought the basic packaging would work, i.e. just put an empty top level init.py, and add a similar file to each of the directories. This didn't work for me.

my_project/
   __init__.py
   src_1/
      __init__.py
      first.py
   src_2/
      __init__.py
      second.py

If I tried to execute second.py from the top level directory I got an Import error. If I setup PYTHONPATH with all 3 directories I got second.py to successfully access first.py

I don't want to use environment variables as the project will almost certainly move, and I don't want to have to configure them for each installation. BTW, I thought that's what packaging does, run top-level init.py first of all and any code in the lower levels should be able to import from sibling directories?

In my attempt to achieve relative paths the top level init.py contains:

import os
import sys

# Get the current directory where this __init__.py file is located
current_directory = os.path.dirname(os.path.abspath(__file__))

# Add project-specific directories to the Python path
project_path = current_directory
src_1_path = os.path.join(current_directory, 'src_1')
src_2_path = os.path.join(current_directory, 'src_2')

sys.path.insert(0, project_path)
sys.path.insert(0, src_1_path)
sys.path.insert(0, src_2_path)

the other 2 init.py files are empty.

first.py contains:

import sys

def print_sys_path():
    print(sys.path)

and second.py contains:

import first

if __name__ == "__main__":
    # bar will be invoked if this module is being run directly, but not via import!
    first.print_sys_path()

When I run this, I get the following error:

D:\my_project>py src_2\second.py Traceback (most recent call last): File "D:\my_project\src_2\second.py", line 1, in import first ModuleNotFoundError: No module named 'first'

My eventual project will have a similar structure, I am just trying to fathom out path independent packaging for it.

It's possible I'm starting off in the wrong direction. I plan to have the following structure on my delivered project:

my_project/
   __init__.py
   my_database/
      __init__.py
      database.py
   my_gui/
      __init__.py
      gui.py
   my_process/
      __init__.py
      process.py

.. and want to eventually package that as 2 SEPARATE executables, gui.exe and processor.exe, using pyinstaller.

Yes, I am a newbie to Python, Packaging etc, so please be gentle. I HAVE read the docs. I cannot find anything relevant to my issue which actually fixes the problem.

Thanks in anticipation for your time and effort!

Solution

  • Greetings sir/madam.


    Unfortunately our fellow boy python uses one of the most annoying packaging systems, So annoying that I had never saw a beginner just starting the concept of packages and not fall into this issue.

    Importing:


    Python importing system unfortunately cannot find files in that particular way, which means when you do try to import first python will search its main package folder for that particular file ( Just like importing any type of libraries, e.x .env, pathlib, sys, ETC ), even if you add it to your python path. ( That is called an absolute import. It would be helpful to learn the differences between relative and absolute import ).

    But you can import the contents of the first file with:

    
import ..src_1.first

    In this relative import, the .. syntax is GOING BACK in the package directory.

    Concept


    Even if you use this type of imports with __init__.py file, python would still raise the same error.

    This is because the files of a package are not meant to run directly, they cannot. In this case, you should not run one of the files, You should run the whole folder.

    python3 -m your_project

    The concept of the __main__.py file in packaging is a long story, you can learn more here.

    Long story short, when you run your whole project folder, python will look through the package to find the __main__.py file. This file must be at the top of your folder hierarchy:

    
your_project/
    __init__.py
    __main__.py

    src_1/
        first.py
        __init__.py

    src_2/
        second.py
        __init__.py

    And that also means that the second.py and first.py files can have relative imports, but the __main__.py file should not go back in the directory and if you want to import any other folders, you should use their absolute path.

    Executing sub-packages:

    Still, if you would like to run sub-packages or treat two packages at a different way although they are bound to each other, you can look it up here.

    Recommendation

    DO NOT USE RELATIVE IMPORTS AND PACKAGING IF NOT NECESSARY

    The Zen of python: If the implementation is hard to explain, it's a bad idea.

    Try using the default packaging system and keep your main file at the top level and avoid using complex import patterns.