How to correctly use python __init__.py in packages

Question

I have such structure lib.

PLUS The setup.py in the root
I want to create a package. How should I correctly set up the init.py in this case. Because after python setup.py sdist and python setup.py install the package is not loaded. And throw an Error.

>>> import test_package
>>> print(test_package.dir2.code.test_code()) 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'test_package' has no attribute 'dir2'
>>> print(test_package.main.test_code_main()) 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'test_package' has no attribute 'main'

I already read docs and dont understand properly how to solve this issue.

The files content:

/test_package/dir1/dir3/__init__.py
None

/test_package/dir1/dir3/code3.py

from test_package.dir1 import code2


def test_code_3():
    return f'{code2.test_code()} Whatsapp!!'

/test_package/dir1/code2.py

from test_package.dir2 import code


def test_code():
    return f'{code.test_code()} dude.'

/test_package/dir1/__init__.py
None

/test_package/dir2/code.py

def test_code():
    return f'Hello'

/test_package/dir2/__init__.py
None

/test_package/main.py

from test_package.dir1.dir3 import code3


def test_code_main():
    return f'{code3.test_code_3()}'

/test_package/__init__.py

__version__ = '0.1.0'

/setup.py

import os
from setuptools import setup
from test_package import __version__

if __name__ == '__main__':
    setup(
        name='test_package',
        python_requires='>=3.11',
        version=os.getenv('PACKAGE_VERSION', __version__),
        packages=['test_package'],
    )

Answer 1

Basically: Existing .py files inside a package structure (nested or not) are not automatically loaded by Python.

The code that is imported and run automatically is the one in __init__.py for each directory inside your package.

Since your test_package/__init__.py file is basically empty (it imports nothing on itself, nor declares any meaningful name but for __version__, importing it also does nothing.

If you want everything in your package structure to be avaliable as soon as test_package is imported, you have to change that file to include:

from . import code2, dir1, dir2, dir3, main

And then either test_package/dir2/__init__.py do perform a from . import code, or keep it empty, and import the contents of dir2 from within the root __init__.py itself with: from .dir2 import code (and the same for dir3, or course).

Since you have the code modules importing one-another, if you import your main.py file, eventually all of them will be imported, and the dotted names will be available as attributes of test_package (that works even if those code imports take place everywhere: the dir2, dir3, attributes are created on the module object that is loaded into memory). That is why your code works sometimes, and sometimes not: in that example you have to explicitly import the main.py file for the chain of imports to be executed (and then we have to follow it like an spaghetti string to be sure everything is imported)

To avoid this irregularity, it is better to put the imports to make sure sub-packages and modules are available in the __init__ file, and just let each other file import the dependencies it really needs into its namespace.