How can I access sibling packages in a maintainable and readable way?

Question

I often end up in a situation where one package needs to use a sibling package. I want to clarify that I'm not asking about how Python allows you to import sibling packages, which has been asked many times. Instead, my question is about a best practice for writing maintainable code.

1. Let's say we have a tools package, and the function tools.parse_name() depends on tools.split_name(). Initially, both might live in the same file where everything is easy:

   # tools/__init__.py
   from .name import parse_name, split_name
   
   # tools/name.py
   def parse_name(name):
     splits = split_name(name)  # Can access from same file.
     return do_something_with_splits(splits)
   
   def split_name(name):
     return do_something_with_name(name)
   

2. Now, at some point we decide that the functions have grown and split them into two files:

    # tools/__init__.py
   from .parse_name import parse_name
   from .split_name import split_name
   
   # tools/parse_name.py
   import tools
   
   def parse_name(name):
     splits = tools.split_name(name)   # Won't work because of import order!
     return do_something_with_splits(splits)
   
   # tools/split_name.py
   def split_name(name):
     return do_something_with_name(name)
   

   The problem is that parse_name.py can't just import the tools package which it is part of itself. At least, this won't allow it to use tools listed below its own line in tools/__init__.py.

3. The technical solution is to import tools.split_name rather than tools:

   # tools/__init__.py
   from .parse_name import parse_name
   from .split_name import split_name
   
   # tools/parse_name.py
   import tools.split_name as tools_split_name
   
   def parse_name(name):
     splits = tools_split_name.split_name(name)   # Works but ugly!
     return do_something_with_splits(splits)
   
   # tools/split_name.py
   def split_name(name):
     return do_something_with_name(name)
   

This solution technically works but quickly becomes messy if more than just one sibling packages are used. Moreover, renaming the package tools to utilities would be a nightmare, since now all the module aliases should change as well.

It would like to avoid importing functions directly and instead import packages, so that it is clear where a function came from when reading the code. How can I handle this situation in a readable and maintainable way?

Answer 1

I'm not proposing this to be actually used in practice, but just for fun, here is a solution using pkgutil and inspect:

import inspect
import os
import pkgutil


def import_siblings(filepath):
  """Import and combine names from all sibling packages of a file."""
  path = os.path.dirname(os.path.abspath(filepath))
  merged = type('MergedModule', (object,), {})
  for importer, module, _ in pkgutil.iter_modules([path]):
    if module + '.py' == os.path.basename(filepath):
      continue
    sibling = importer.find_module(module).load_module(module)
    for name, member in inspect.getmembers(sibling):
      if name.startswith('__'):
        continue
      if hasattr(merged, name):
        message = "Two sibling packages define the same name '{}'."
        raise KeyError(message.format(name))
      setattr(merged, name, member)
  return merged

The example from the question becomes:

# tools/__init__.py
from .parse_name import parse_name
from .split_name import split_name

# tools/parse_name.py
tools = import_siblings(__file__)

def parse_name(name):
  splits = tools.split_name(name)  # Same usage as if this was an external module.
  return do_something_with_splits(splits)

# tools/split_name.py
def split_name(name):
  return do_something_with_name(name)