class hidden from module dictionary

Question

I'm developing a code analysis tool for Python program.
I'm using introspection techniques to navigate into program structure.
Recently, I tested my tool on big packages like tkinter and matplotlib. It worked well.
But I found an oddity when analyzing numpy.

import numpy,inspect
for elem in inspect.getmembers( numpy, inspect.isclass)
  print( elem)
print( 'Tester' in dir( numpy))
print( numpy.__dict__['Tester'])

Result:

blablabla
('Tester', <class 'numpy.testing._private.nosetester.NoseTester'>),
blablabla

True
KeyError: 'Tester'

getmembers() and dir() agree that there is a 'Tester' class but it is not in __dict__ dictionary. I dug a little further:

1  >>> import numpy,inspect
2  >>> d1 = inspect.getmembers( numpy)
3  >>> d2 = dir( numpy)
4  >>> d3 = numpy.__dict__.keys()
5  >>> len(d1),len(d2),len(d3)
6  (602, 602, 601)
7  >>> set([d[0] for d in d1]) - set(d3)
8  {'Tester'}
9  numpy.Tester
10 <class 'numpy.testing._private.nosetester.NoseTester'>
11 >>> 

getmembers() and dir() agree but __dict__ do not. Line 8 shows that 'Tester' is not in __dict__.
This bring questions:

• what is the mechanism used by numpy to hide the 'Tester' class?
• where are getmembers() and dir() finding the reference to 'Tester' class?

I'm using Python 3.9.2 and numpy 1.23.5

Answer 1

I believe inspect.getmembers relies on dir of an object for the keys, and getattr for the values, and dir for the numpy class is overridden to:

  def __dir__():
    return list(globals().keys() | {'Tester', 'testing'})

with the getattr overridden specifically in regard to the above to:

        if attr == 'testing':
            import numpy.testing as testing
            return testing
        elif attr == 'Tester':
            from .testing import Tester
            return 

so dir will return a "Tester", and getattr will find and return a corresponding object, but it's not in the __dict__.

This is the reasoning they use is to allow for a lazy import:

        # Importing Tester requires importing all of UnitTest which is not a
        # cheap import Since it is mainly used in test suits, we lazy import it
        # here to save on the order of 10 ms of import time for most users
        #
        # The previous way Tester was imported also had a side effect of adding
        # the full `numpy.testing` namespace

numpy dir definition

numpy getattr

getmembers definition

Example

>>> import numpy as np
>>> import inspect
>>> 
>>> np.__dir__ = lambda: ["poly"]
>>> 
>>> dir(np)
['poly']
>>>
>>> inspect.getmembers(np)
[('poly', <function poly at 0x101fd8280>)]
>>> 

if you override getattr as well, then you can create that "hidden" attribute:

>>> import numpy as np
>>> import inspect
>>> 
>>> np.__dir__ = lambda: ["this_doesnt_exist","poly"]
>>> 
>>> "this_doesnt_exist" in np.__dict__
False
>>> "poly" in np.__dict__
True
>>> 
>>> inspect.getmembers(np) # this_doesnt_exist neither in dict, or successfully returned from getattr
[('poly', <function poly at 0x105ccc280>)]
>>> 
>>> np.__getattr__ = lambda x: f"{x} doesnt exist, but my getattr pretends it does."
>>> 
>>> inspect.getmembers(np)
[('poly', <function poly at 0x105ccc280>), ('this_doesnt_exist', 'this_doesnt_exist doesnt exist, but my getattr pretends it does.')]
>>>