Python how to print full stack, including magic methods (dunder methods) used?

Question

I am trying to debug a Python built-in class. My debugging has brought me into the realm of magic methods (aka dunder methods).

I am trying to figure out which dunder methods are called, if any. Normally I would do something like this:

import sys
import traceback

# This would be located where the I'm currently debugging
traceback.print_stack(file=sys.stdout)

However, traceback.print_stack does not give me the level of detail of printing what dunder methods area used in its vicinity.

Is there some way I can print out, in a very verbose manner, what is actually happening inside a block of code?

---

Sample Code

#!/usr/bin/env python3.6

import sys
import traceback
from enum import Enum


class TestEnum(Enum):
    """Test enum."""

    A = "A"


def main():
    for enum_member in TestEnum:
        traceback.print_stack(file=sys.stdout)
        print(f"enum member = {enum_member}.")


if __name__ == "__main__":
    main()

I would like the above sample code to print out any dunder methods used (ex: __iter__).

Currently it prints out the path to the call to traceback.print_stack:

/path/to/venv/bin/python /path/to/file.py
  File "/path/to/file.py", line 56, in <module>
    main()
  File "/path/to/file.py", line 51, in main
    traceback.print_stack(file=sys.stdout)
enum member = TestEnum.A.

---

P.S. I'm not interested in going to the byte code level given by dis.dis.

Answer 1

I think, with the stacktrace you are looking at the wrong place. When you call print_stack from a place, that is executed only when coming from a dunder method, this method is very well included in the output.

I tried this code to verify:

import sys
import traceback
from enum import Enum


class TestEnum(Enum):
    """Test enum."""

    A = "A"


class MyIter:

    def __init__(self):
        self.i = 0

    def __next__(self):
        self.i += 1
        if self.i <= 1:
            traceback.print_stack(file=sys.stdout)
            return TestEnum.A
        raise StopIteration

    def __iter__(self):
        return self


def main():
    for enum_member in MyIter():
        print(f"enum member = {enum_member}.")


if __name__ == "__main__":
    main()

The last line of the stack trace is printed as

File "/home/lydia/playground/demo.py", line 21, in __next__
traceback.print_stack(file=sys.stdout)

In your original code, you are getting the stack trace at a time when all dunder methods have already returned. Thus they have been removed from the stack.

So I think, you want to have a look at a call graph instead. I know that IntelliJ / PyCharm can do this nicely at least in the paid editions.

There are other tools that you may want to try. How does pycallgraph look to you?

Update:

Python makes it actually pretty easy to dump a plain list of all the function calls.

Basically all you need to do is

import sys
sys.setprofile(tracefunc)

Write the tracefunc depending on your needs. Find a working example at this SO question: How do I print functions as they are called

Warning: I needed to start the script from an external shell. Starting it by using the play button in my IDE meant that the script would never terminate but write more and more lines. I assume it collides with the internal profiling done by my IDE.

The official documentation of sys.setprofile: https://docs.python.org/3/library/sys.html#sys.setprofile

And a random tutorial about tracing in Python: https://pymotw.com/2/sys/tracing.html

Note however, that by my experience you can get the best insights into the questions "who is calling whom?" or "where does this value even come from?" by using a plain-old debugger.