I am trying to understand the process by which python code gets executed. Suppose the source has a function definition. Using ast.parse(), I parse it into an ast, which will contain an instance of the FunctionDef node class. This node instance is not a callable and is not the same as the function object. How can the function object, with all its dunder attributes, be created from this ast?
You can't (as far as I know) compile an arbitrary individual AST node like a FunctionDef. What you can do is compile the entire code snippet as a module, exec it in a provided namespace, and then access its contents, including the function. Here's an example:
import ast
txt = """
def foo(x, y=2):
z = x*y + 3
print("z is", z)
return z**2
"""
tree = ast.parse(txt, mode='exec')
code = compile(tree, filename='blah', mode='exec')
namespace = {}
exec(code, namespace)
Now namespace is the equivalent of the __dict__ of a module containing the given code. You can access and call the function:
>>> namespace['foo']
<function foo at 0x00000000023A2B70>
>>> namespace['foo'](2, 3)
z is 9
81
Note that if this is all you want to do, there's no need to use ast at all. You can just compile the source string directly with compile(tree, filename='blah', mode='exec'). And in fact there's no need to even involve compile, since you can just exec the source string directly with exec(txt, namespace). If your goal is just to get the final function object out, you don't really need access to the internal parse and compile steps; just exec the whole thing in a namespace and then grab the function from there.