Calling a common function across multiple files in a directory in Python

Question

I have a python project with the following structure

- my-project/
  - my_project/
    - __init__.py
    - modules_charts/
      - chart_a.py
      - chart_b.py
      - ...
    - main.py
  - poetry.lock
  - pyproject.toml
  - README.md

My goal is to have a dynamic modules_charts folder that contain a non-limited list of python files with non-standard naming. Each file contain the same function called create_chart().

I want to be able to call all those create_chart() functions from the main.py script.

I want to avoid updating main.py every time I add a new file within the modules_charts folder.

Until now I was using the following code in main.py:

for file_name in sorted(os.listdir("modules_charts")):
    if not file_name.endswith(".py"):
        continue
    module_name = file_name[:-3]
    module = import_module("." + module_name, package="modules_charts")
    module.main()

I have 2 concerns about this working solution:

1. It generates an issue if I am running my script with python my_project/main.py
2. There might be another way of doing it by using import and __init__.py

What would be the proper way to achieve this without taking into account the directory where I am running my script.

Answer 1

You likely want to use __init__.py to help expose sub modules.

main.py

import inspect
import modules_charts

## call every member's create_chart method
for member in inspect.getmembers(modules_charts, inspect.ismodule):
    getattr(modules_charts, member[0]).create_chart()

modules_charts/__init__.py

## directly expose these modules by name
from . import chart_a
from . import chart_b

modules_charts/chart_a.py

def create_chart():
    print("hello")

modules_charts/chart_b.py

def create_chart():
    print("world")

This should give you:

hello
world

Of course you need to update __init__.py when you want to directly expose new charts but that is part of your chart infrastructure so not as big a problem as keeping uses of the charting up to date.

To further simplify main.py you might write a method in __init__.py to return the modules. Maybe something like:

main.py

import modules_charts

## call every member's create_chart method
for chart in modules_charts.get_charts():
    chart.create_chart()

modules_charts/__init__.py

import types
from . import chart_a
from . import chart_b

def get_charts():
    for name, value in globals().items():
        if isinstance(value, types.ModuleType) and name != "types":
            yield value

That should give you the same result when main.py is run.

If you wanted to leverage __all__ in your module, you can also do this.

modules_charts/__init__.py

from . import chart_a
from . import chart_b

__all__ = ["chart_a", "chart_b"]

That simplifies your module quite a bit, but it might make it a little harder to use it as:

import modules_charts

for chart_name in modules_charts.__all__:
    getattr(modules_charts, chart_name).create_chart()