Python — Init class reflectively

Question

I am creating a commands system in Python. I have module vkcommands that has a class that processes commands from chat (this is a chat-bot), and inside it, I also have class VKCommand with attributes like name, usage, min_rank, etc. Then I have module vkcmds with submodules that implement these commands:

...
vkcommands.py
vkcmds
    |- __init__.py  # empty
    |- add_group.py
    |- another_cmd.py
    |- ...

Implementations of commands (e.g. add_group) look like this:

import ranks
import vkcommands
from vkcommands import VKCommand


class AddGroup(VKCommand):
    def __init__(self, kristy):
        VKCommand.__init__(self, kristy,
                           label='create',
                           # ... (other attributes)
                           min_rank=ranks.Rank.USER)

    def execute(self, chat, peer, sender, args=None, attachments=None):
        # implementation (called from vkcommands.py)

When a user sends a message in the chat, the command manager analyzes it and looks through the registered commands list to see if this is an ordinary message or a bot command. Currently I register all commands in the commands list manually like this:

class VKCommandsManager:
    def __init__(self, kristy):
        from vkcmds import (
            add_group,
            next_class
        )

        self.kristy = kristy
        self.commands = (
            add_group.AddGroup(kristy),
            next_class.NextClass(kristy)
        )

Now I would like all commands to be registered automatically using reflections instead. In Java, I'd iterate over all classes in my commands package, reflectively getConstructor of each, call it to retrieve the VKCommand object, and add it to the commands list.

How can I do so in Python? Again, what I need is to:

1. iterate over all submodules in module (folder) vkcmds/;
2. for each submodule, check if there is some class X that extends VKCommand inside;
3. if (2) is true, then call the constructor of that class with one argument (it is guaranteed that the constructor for all commands only has one argument of a known type (my bot's main class));
4. add the object (? extends VKCommand) constructed in (3) to the commands list that I can iterate over later.

Answer 1

With this file structure:

- Project
   ├─ commands
   |   ├─ base.py
   |   ├─ baz.py
   |   └─ foo_bar.py
   |
   └─ main.py

And the following inside the commands directory files:

• base.py

  class VKCommand:
      """ We will inherit from this class if we want to include the class in commands.  """
  

• baz.py

  from commands.base import VKCommand
  
  class Baz(VKCommand):
      pass
  
  
  def baz():
      """ Random function we do not want to retrieve.  
  

• foo_bar.py

  from .base import VKCommand
  
  
  class Foo(VKCommand):
      """ We only want to retrieve this command.  """
      pass
  
  
  class Bar:
      """ We want to ignore this class.  """
      pass
  
  
  def fizz():
      """  Random function we do not want to retrieve. """
  

We can retrieve the class instances and names directly using the following code:

• main.py

  """
    Dynamically load all commands located in submodules.
    This file is assumed to be at most 1 level higher than the
    specified folder.
  """
  
  import pyclbr
  import glob
  import os
  
  def filter_class(classes):
      inherit_from = 'VKCommand'
      classes = {name: info for name, info in classes.items() if inherit_from in info.super}
      return classes
  
  # Locate all submodules and classes that it contains without importing it.
  folder = 'commands'  # `vkcmds`.
  submodules = dict()
  absolute_search_path = os.path.join(os.path.dirname(__file__), folder, '*.py')
  for path in glob.glob(absolute_search_path):
      submodule_name = os.path.basename(path)[:-3]
      all_classes = pyclbr.readmodule(f"commands.{submodule_name}")
      command_classes = filter_class(all_classes)
      if command_classes:
          submodules[submodule_name] = command_classes
  
  # import the class and store an instance of the class into the command list
  class_instances = dict()
  for submodule_name, class_names in submodules.items():
      module = __import__(f"{folder}.{submodule_name}")
      submodule = getattr(module, submodule_name)
      for class_name in class_names:
          class_instance = getattr(submodule, class_name)
          class_instances[class_name] = class_instance
  
  print(class_instances)
  

Explanation

The solution is twofold. It first locates all submodules that have a class which inherit from VKCommand and are located in the folder 'commands`. This leads to the following output containing the module and the class that have to be imported and instantiated respectively:

{'baz': {'Baz': <pyclbr.Class object at 0x000002BF886357F0>}, 'foo_bar': {'Foo': <pyclbr.Class object at 0x000002BF88660668>}}

The second part of the code imports the correct module and class name at run time. The variable class_instance contains the class name and a reference to the class which can be used to instantiate it. The final output will be:

{'Baz': <class 'commands.baz.Baz'>, 'Foo': <class 'commands.foo_bar.Foo'>}

Important notes:

1. The code only works when importing modules that are 1 dictionary deeper. If you want to use it recursively, you will have to locate the relative path difference and update the pyclbr.readmodule and __import__ with the correct (full) relative import path.

2. Only the modules that contain a class which inherit from VKCommand get loaded. All other modules are not imported, and have to be imported manually.