I'm a bit confused with working with remote third-party libraries:
1) For example, I have server code:
import Pyro4
import Pyro4.naming
import Pyro4.utils.flame
Pyro4.config.REQUIRE_EXPOSE = False
Pyro4.config.FLAME_ENABLED = True
Pyro4.config.SERIALIZERS_ACCEPTED = set(["pickle"])
Pyro4.config.SERIALIZER = 'pickle' # for flameserver
def _main():
"""Start RPC server."""
ip_address = Pyro4.socketutil.getIpAddress(None, workaround127=True)
ns_daemon = Pyro4.naming.NameServerDaemon(host=ip_address, port=RPC_PORT)
Pyro4.utils.flame.start(ns_daemon)
from remote_management.rpc.component.shell import Shell
shell_uri = ns_daemon.register(Shell())
ns_daemon.nameserver.register("shell", shell_uri)
from remote_management.rpc.component.cpu import CPU
cpu_uri = ns_daemon.register(CPU())
ns_daemon.nameserver.register("cpu", cpu_uri)
ns_daemon.requestLoop()
ns_daemon.close()
if __name__ == "__main__":
_main()
The CPU component is like this:
import psutil
class CPU(object):
def ps(self):
return psutil.cpu_times()
When I use it in client like this:
import Pyro4
import Pyro4.utils.flame
import Pyro4.errors
Pyro4.config.SERIALIZER = "pickle"
proxy = Pyro4.Proxy("PYRONAME:cpu@myhost:47976")
print proxy.ps()
I get an error:
Traceback (most recent call last):
File "t.py", line 145, in <module>
print proxy.ps()
File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/core.py", line 187, in __call__
return self.__send(self.__name, args, kwargs)
File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/core.py", line 464, in _pyroInvoke
data = serializer.deserializeData(msg.data, compressed=msg.flags & message.FLAGS_COMPRESSED)
File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/util.py", line 170, in deserializeData
return self.loads(data)
File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/util.py", line 451, in loads
return pickle.loads(data)
ImportError: No module named psutil._pslinux
As you see it cannot be deserialized. And if I change return to:
return tuple(psutil.cpu_times()), then it works.
2) I think this next problem has the same nature: The server code is almost the same except component
from remote_management.rpc.component.registrator import ModuleRegistrator
reg_uri = ns_daemon.register(ModuleRegistrator())
ns_daemon.nameserver.register("module_registrator", reg_uri)
If I need remote module, but don't have it locally, why I can't do just like this:
Registrator component:
class ModuleRegistrator(object):
def register(self, module):
module = importlib.import_module(module)
self._pyroDaemon.register(module)
return module
so when I use it in client:
proxy = Pyro4.Proxy("PYRONAME:module_registrator@myhost:47976")
print proxy.register("psutil").cpu_times()
I get an error:
Traceback (most recent call last):
File "t.py", line 145, in <module>
print proxy.register('psutil').cpu_times()
File "/home/korolev/.envs/auto/lib/python2.7/site-packages/Pyro4/core.py", line 279, in __getattr__
raise AttributeError("remote object '%s' has no exposed attribute or method '%s'" % (self._pyroUri, name))
AttributeError: remote object 'PYRONAME:module_registrator@myhost:47976' has no exposed attribute or method 'register'
Please help me understand this errors and where I'm missing something? And what is the best approach to handle things like this when working with third-party remotely? Thanks in advance!
Firstly, you seem to be using the 'flame' feature of Pyro4, which requires the pickle serialization protocol. This is powerful, but not without problems.
Are you sure you want/need to use flame? I hope you are aware of its security implications. http://pyro4.readthedocs.io/en/latest/flame.html
Now, your actual question consists of three parts (try to make your future questions more about just a single issue, that makes answering them a lot simpler)
As you already found out yourself, you cannot assume that all types can just be transported over the network and magically just work on the other side. Especially with pickle this only works if the module in which
the type is defined, is available on both sides.
In your case, it seems that the psutil module is not available on your client side, hence the pickle error when it tries to deserialize the object returned by psutil.cpu_times().
The solution could be to install psutil on your client side as well.
You also discovered an alternative solution: don't pass objects of custom types over the wire, but stick to builtin types as much as possible. When you convert the result to a tuple or list or dict,
these can be serialized and deserialized without any issues (if all the objects they contain are of a builtin type as well).
This code looks like you're trying to find a workaround for the problem in the first question?
Anyway, the error should be clear; it means what it says: that the remote object your proxy is connected to, doesn't have the method exposed that you're trying to call. You should add a @Pyro4.expose decoration
to properly expose it.
That said, you're already using Flame, so why aren't you using its remote module feature? See http://pyro4.readthedocs.io/en/latest/flame.html#flame-object-and-examples
For example (I assume you installed psutil on the client side):
import Pyro4.utils.flame
Pyro4.config.SERIALIZER = "pickle"
flame = Pyro4.utils.flame.connect("yourhost:9999")
psutilmodule = flame.module("psutil")
print("remote cpu:", psutilmodule.cpu_times())
to address your question "what is the best approach to handle things like this when working with third-party remotely":