Hook the global name lookup in a python interpreter

Question

Here is the thing, I have a proxy holding the reference to a remote module, and I put some of these proxies to the sys.modules such that I can use it just like local modules. But some other objects are put in the __builtin__ module at the remote environment (like a magic variable for convenience of debugging or referencing). I don't want to reference these vars like conn.__builtin__.var, and I have to either replace the local __builtin__ (which seems not working for replace sys.modules['__builtin__'] or to hook the global name finding rules. How? For a module you can just overload a getattr to do this. But in a interactive interpreter like IPython, who is the main module or how to do this? update: As pointed out by @Nizam Mohamed, yes I can get the __main__ module, but still I can't modify the name lookup role of it.

I'd like to turn the local environment completely to be the remote one (for a debugging console)

UPDATE

For now I just iterate all the __builtin__.__dict__ and if there is a name that isn't in the local __builtin__. I add the name to local's __builtin__. But it's not so dynamic compare to a name lookup rule say if I can't find the name in local __builtin__ try the remote one.

here is a similar discussion.

And this question gives a simulation of module by replace it with a object in sys.modules. But this won't work for __builtin__ name lookup, I've also tried to replace the __builtin__.__getattribute__ with a custom one that will first use the original lookup followed by a custom one when failed. But global name lookup of __builtin__ never called into the __builtin__.__getattribute__ even __builtin__.__getattribute__('name') returns the desired value, __builtin__.name or name never returns one.

Answer 1

Use AST transformation of IPython shell

As @asmeurer said, you can write a simple AST transformer to "hook" the variable name lookup. The base class ast.NodeTransformer provide a visit_Name method that you can manipulate. You just need to overload this method to redefine those variables existing in the remote module but not locally.

The following module can be used as an IPython extension:

testAST.py

import ast

modName = "undefined"
modAttr = []
user_ns = {}
class MyTransformer(ast.NodeTransformer):
    def visit_Name(self, node):
        if node.id in modAttr and not node.id in user_ns: 
          return self.getName(node)
        return node
    def getName(self, NameNode):
        return ast.Attribute(value=ast.Name(id=modName, ctx=ast.Load()), 
                             attr = NameNode.id, 
                             ctx  = NameNode.ctx)
def magic_import(self, line):
    global modName, modAttr, user_ns
    modName = str(line)
    if not self.shell.run_code( compile('import {0}'.format(line), '<string>', 'exec') ):
       user_ns = self.shell.user_ns
       modAttr = user_ns[line.strip()].__dict__
       self.shell.ast_transformers.append(MyTransformer())
       print modName, 'imported'
    
def load_ipython_extension(ip):
    ip.define_magic('magic_import', magic_import)

dummyModule.py

robot=" World"

Usage:

In [1]: %load_ext testAST
In [2]: %magic_import dummyModule
In [3]: print "Hello" , robot
Hello World

In [4]: dummyModule.robot_II = "Human" 
In [5]: print "Hi", robot_II
Hi Human

The benefit of this method is that any modification to the remote module takes effect immediately because the lookup is done in the language level and no object is copied and cached.

One drawback of this method is not being able to handle dynamic lookup. If that's important for you, maybe the python_line_transforms hook is more suitable.