How to merge dependent globals from multiple calls to exec with independent globals dicts

This code works fine - it defines do_return as a global, it defines do_proxy as a global, and do_proxy can resolve do_return when called.

g1 = {}
exec("def do_return(): return 1", g1)
exec("def do_proxy(): return do_return()", g1)
merged = {**g1}
merged.pop("__builtins__")
print(merged.keys())
exec("print(do_proxy())", merged)

$ python3 main.py 
dict_keys(['do_return', 'do_proxy'])
1

This code doesn't:

g1 = {}
g2 = {}
exec("def do_return(): return 1", g1)
exec("def do_proxy(): return do_return()", g2)
merged = {**g1, **g2}
merged.pop("__builtins__")
print(merged.keys())
exec("print(do_proxy())", merged)

It complaints that do_return is not defined, even though there is a global named do_return:

$ python3 main.py 
dict_keys(['do_return', 'do_proxy'])
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    exec("print(do_proxy())", merged)
  File "<string>", line 1, in <module>
  File "<string>", line 1, in do_proxy
NameError: name 'do_return' is not defined

Is there a way to merge dependent globals from multiple calls to exec which didn't share the same globals dict when they were originally called?

The same happens if I call global do_return inside do_proxy (though if that was required, I would also expect the first example to fail, similarly requiring it).

Solution

When you call a function, it uses its own reference to the global namespace where it was defined, rather than the globals of where it is being called from. You can see this by checking the __globals__ attribute of a function. In the first version of your code, merged['do_proxy'].__globals__ is a reference to g1, which also contains your do_return function. But in the second version of your code, do_proxy's __globals__ is a reference to g2, which does not contain the other function.

This makes sense if you think about it. It's similar to how a function defined in another module will look for names in its module's global namespace, not in the namespace where you've imported it, and functions in two different modules can only interact if you pass references around after importing them, or if one of the modules imports the other.

There might be some workarounds you could use in your code. One option would be to merge the g1 definitions into the g2 dictionary, rather than creating a new dict with the combined results.

g2.update(g1) # instead of creating merged

Of course, this works for do_proxy referring to do_return, but it wouldn't work in reverse, if do_return wanted to call do_proxy (perhaps as part of some kind of recursive algorithm). You could perhaps merge both ways, but that would probably get messy.