I created a system written in Python that provides various hardware control, data acquisition, and processing functions. I want this system to be accessible from programs written in Python or other languages, like Lua, Matlab, Octave, C++ and possibly others. Additionally, those programs may be running either on that computer or on another machine connected via a network.
I'm aware of the existence of various RPC libraries for Python like
Is there available any simple and lightweight solution enabling remote calling of Python functions from programs written in other languages?
I have developed a similar system in 2012, where the arguments to the Python functions and returned values were passed via MessagePack. Such a solution enables communication with programs written in various languages. It was published in the Usenet alt.sources group https://groups.google.com/g/alt.sources/c/QasPxJkKIUs (also available in funet archive http://ftp.funet.fi/pub/archive/alt.sources/2722.gz ). Now I have supplemented it with ZeroMQ library that provides efficient communication with programs running either on the same or on remote machines.
The core of the system is the server written in Python:
#!/usr/bin/python3
"""
The code below implements a simple ZeroMQ and MessagePack RPC server.
Written by Wojciech M. Zabolotny, wzab<at>ise.pw.edu.pl or wzab01<at>gmail.com
13.04.2021
The code is based on a post: "Simple object based RPC for Python",
https://groups.google.com/g/alt.sources/c/QasPxJkKIUs/m/An1JnLooNCcJ
https://www.funet.fi/pub/archive/alt.sources/2722.gz
This code is published as PUBLIC DOMAIN or under
Creative Commons Zero v1.0 Universal license (whichever better suits your needs).
"""
import time
import zmq
import msgpack as mp
import traceback
import os
#Functions which process requests
def remote_mult(a,b):
return a*b
def remote_div(a,b):
print(a,b,a/b)
return a/b
def cat_file(fname):
f=open(fname,"rb")
return f.read()
#Table of functions
func={
'mult':remote_mult,
'div':remote_div,
'file':cat_file,
}
def handle(msg):
try:
obj = mp.unpackb(msg)
if len(obj) != 2:
raise Exception("Wrong number of RPC objects, should be 2: name and arguments")
if isinstance(obj[1],tuple) or isinstance(obj[1],list):
res=func[obj[0]](*obj[1])
elif isinstance(obj[1],dict):
res=func[obj[0]](**obj[1])
else:
raise Exception("Wrong type of arguments in RPC, should be list, tuple or dictionary")
res = ("OK", res)
except Exception:
res=("error", traceback.format_exc())
return mp.packb(res)
if __name__ == "__main__":
context = zmq.Context()
socket = context.socket(zmq.REP)
# Create the server, binding to localhost on port 9999
socket.bind("tcp://*:9999")
while True:
msg = socket.recv()
res = handle(msg)
socket.send(res)
The server may publish different functions with different arguments. The arguments may be given either as positional arguments (then they should be passed in a tuple) or as keyword arguments (then they should be passed as a map/dictionary).
An example of Python client using those functions is given below:
#!/usr/bin/python3
"""
The code below implements a simple ZeroMQ and MessagePack RPC client.
Written by Wojciech M. Zabolotny, wzab<at>ise.pw.edu.pl or wzab01<at>gmail.com
13.04.2021
The code is based on a post: "Simple object based RPC for Python",
https://groups.google.com/g/alt.sources/c/QasPxJkKIUs/m/An1JnLooNCcJ
https://www.funet.fi/pub/archive/alt.sources/2722.gz
This code is published as PUBLIC DOMAIN or under
Creative Commons Zero v1.0 Universal license (whichever better suits your needs).
"""
import socket
import sys
import msgpack as p
import zmq
HOST, PORT = "localhost", 9999
data = " ".join(sys.argv[1:])
objects=[
["mult",(4,5)],
["mult",{"a":7,"b":8}],
["div",{"a":9,"b":4}],
["file",("/etc/passwd",)],
["file",("/etc/non_existing_file",)],
]
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:9999")
for obj in objects:
socket.send(p.packb(obj))
# Get the reply.
msg = socket.recv()
resp = p.unpackb(msg)
print("Received reply", resp)
Below are the clients written in other languages.
In Lua:
-- Demonstrator of the communication with simple Python RPC server from Lua
-- Written by Wojciech M. Zabołotny (wzab01<at>gmail.com or wzab<at>ise.pw.edu.pl)
-- Copyright: This program is released into the public domain or under
-- Creative Commons Zero v1.0 Universal license (whichever better suits your needs).
local zmq = require "lzmq"
--require "utils"
local mp = require "mpack"
--print_version(zmq)
local pr = require "pl.pretty"
context = assert(zmq.context())
rpcsrv = assert(context:socket (zmq.REQ))
assert(rpcsrv:connect("tcp://localhost:9999"))
function rpc(params)
local req=mp.pack(test)
rpcsrv:send(req)
local rcv=rpcsrv:recv()
local res=mp.unpack(rcv)
return res
end
test = {"file",{"/etc/passwd"}}
local res = rpc(test)
pr.dump(res)
test = {"mult",{7,8}}
res = rpc(test)
pr.dump(res)
test = {"div",{b=4.0,a=9.0}}
res = rpc(test)
pr.dump(res)
-- The above works, but 9/4 is printed as 2.
print(res[2])
test = {"div",{a=4.0,b=9.0}}
res = rpc(test)
pr.dump(res)
-- The above works, but 4/9 is printed as 0.
print(res[2])
in Octave
% Demonstrator of the communication with simple Python RPC server from Octave
% Written by Wojciech M. Zabołotny (wzab01<at>gmail.com or wzab<at>ise.pw.edu.pl)
% Copyright: This program is released into the public domain.
pkg load jsonlab
pkg load zeromq
srv = zmq_socket (ZMQ_REQ);
zmq_connect (srv, "tcp://localhost:9999");
function res = rpc(req,fname,fargs,maxlen=10000)
x={fname, fargs};
a=savemsgpack(x);
zmq_send(req,a);
w=zmq_recv(req,maxlen);
y=loadmsgpack(char(w));
if strcmp(char(y{1}),"OK")
res = y{2};
end
if strcmp(char(y{1}),"error")
error(char(y{2}));
end
endfunction
res = rpc(srv,"mult",struct("a",13,"b",20));
res
res = rpc(srv,"mult",{17,3});
res
res = rpc(srv,"file",{"/etc/passwd"});
char(res')
and finally in C++
// Demonstrator of the communication with simple Python RPC server from C++
// Written by Wojciech M. Zabołotny (wzab01<at>gmail.com or wzab<at>ise.pw.edu.pl)
// Copyright: This program is released into the public domain or under
// Creative Commons Zero v1.0 Universal license (whichever better suits your needs).
#include <string>
#include <zmq.hpp>
#include <iostream>
#include <msgpack.hpp>
msgpack::object_handle rpc(zmq::socket_t &sock, auto req)
{
std::size_t offset = 0;
std::stringstream rstr;
msgpack::pack(rstr,req);
zmq::message_t msg(rstr.str());
sock.send(msg,zmq::send_flags::none);
auto res = sock.recv(msg, zmq::recv_flags::none);
auto oh = msgpack::unpack((const char *)msg.data(),msg.size(),offset);
return oh;
}
int main(void) {
zmq::context_t ctx;
zmq::socket_t sock(ctx, zmq::socket_type::req);
sock.connect("tcp://localhost:9999");
msgpack::object_handle res;
res = rpc(sock,std::tuple<std::string, std::array<int,2>>({"mult", {7, 8}}));
std::cout << res.get() << std::endl;
res = rpc(sock,std::tuple<std::string, std::map<std::string,int>>({"div", {{"a",8},{"b",7}}}));
std::cout << res.get() << std::endl;
res = rpc(sock,std::tuple<std::string, std::map<std::string,int>>({"div", {{"b",8},{"a",7}}}));
std::cout << res.get() << std::endl;
res = rpc(sock,std::tuple<std::string, std::tuple<std::string>>({ "file", {"/etc/passwd"}}));
std::cout << res.get() << std::endl;
}
The above C++ code must be compiled with the -fconcepts-ts option e.g. like below:
c++ obj_rpc_cli.cc -o obj_rpc_cli -fconcepts-ts -g -lzmq -lmsgpackc
The solution is really minimal. The maintained version is available in the github repository: https://gitlab.com/WZab/python-versatile-rpc