Search code examples
pythonsocketsserializationserverpickle

How to send custom Python objects through sockets without having a reference to object's class in other side of socket?

I am trying to send a custom Python object from a client to a server, but the additional challenge of not having the reference to the object's class in the server. If you have the class declaration in the server script, then it reconstructs the Python object correctly. Otherwise, it just fails to reconstruct.

I have tried using pickle and jsonpickle to send the Python object (you can select both options through the OPTION variable). Both methods do not work after commenting out the class declaration.

Server

from socket import socket, gethostbyname, AF_INET, SOCK_DGRAM
import pickle

import jsonpickle

OPTION = 'pickle'
PORT_NUMBER = 5000
SIZE = 4000

# Required (but I don't want this!)
# class A:
#     ...

s = socket(AF_INET, SOCK_DGRAM)
s.bind((gethostbyname('0.0.0.0'), PORT_NUMBER))

(data,addr) = s.recvfrom(SIZE)
print(data)

if OPTION == 'pickle':
    output = pickle.loads(data)
    print(output)

elif OPTION == 'jsonpickle':
    output = jsonpickle.decode(data)
    print(output)

Client

import sys
from socket import socket, AF_INET, SOCK_DGRAM
import pickle

import jsonpickle

OPTION = 'pickle'
SERVER_IP   = '127.0.0.1'
PORT_NUMBER = 5000

class A:
    ...

a = A()

s = socket(AF_INET, SOCK_DGRAM)
s.connect((SERVER_IP, PORT_NUMBER))

if OPTION == 'pickle':
    s.send(pickle.dumps(a))
elif OPTION == 'jsonpickle':
    s.send(jsonpickle.encode(a).encode())

Outputs

These are the outputs when class A is in the server:

$ python server.py # jsonpickle
b'{"py/object": "__main__.A"}'
<__main__.A object at 0x7f4557a675e0>
$ python server.py # pickle
b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.'
<__main__.A object at 0x7fcab9fc9bb0>

These are the outputs when class A is not part of the server:

$ python server.py # pickle
b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94.'
Traceback (most recent call last):
  File "/home/eduardo/Downloads/SO_Question/server.py", line 21, in <module>
    output = pickle.loads(data)
AttributeError: Can't get attribute 'A' on <module '__main__' from '/home/eduardo/Downloads/SO_Questi
on/server.py'>
$ python server.py # jsonpickle
b'{"py/object": "__main__.A"}'
{'py/object': '__main__.A'}

I looked into other approaches, like Pyro, but I am not too sure if these will work for my use case. Maybe ther are other tools that could help here. This server-client example is just a toy problem to a larger project I am working on: ChimeraPy. The goal would be for a server to distribute custom and unique code to clients. By using clients' computational resources to execute the custom code, using proxies, like Pyro, are not ideal as they don't distribute the computational load the same way as a cluster does.

Any help would be greatly appreciated. Thanks!

Tried:

Hoping:

  • To find a solution for sending custom Python objects without reference in the other side of the socket
Solution

  • Using the recommend links 1 and 2 by @metatoaster, I was able to achieve the desired behavior:

    Server

    from socket import socket, gethostbyname, AF_INET, SOCK_DGRAM
import pickle
import dill

import jsonpickle

OPTION = 'dill'
PORT_NUMBER = 5000
SIZE = 4000

# Required (but I don't want this!)
# class A:
#     ...

s = socket(AF_INET, SOCK_DGRAM)
s.bind((gethostbyname('0.0.0.0'), PORT_NUMBER))

(data,addr) = s.recvfrom(SIZE)
print(data)

if OPTION == 'pickle':
    output = pickle.loads(data)
    print(output)

elif OPTION == 'jsonpickle':
    output = jsonpickle.decode(data)
    print(output)

elif OPTION == 'dill':
    output = dill.loads(data)
    print(output)
    print(output.multiple(3))

    Client

    import sys
from socket import socket, AF_INET, SOCK_DGRAM
import pickle

import jsonpickle
import dill

OPTION = 'dill'
SERVER_IP   = '127.0.0.1'
PORT_NUMBER = 5000

class A:
    def __init__(self):
        self.num = 2

    def multiple(self, n):
        return self.num * n

a = A()

s = socket(AF_INET, SOCK_DGRAM)
s.connect((SERVER_IP, PORT_NUMBER))

if OPTION == 'pickle':
    s.send(pickle.dumps(a))
elif OPTION == 'jsonpickle':
    s.send(jsonpickle.encode(a).encode())
elif OPTION == 'dill':
    s.send(dill.dumps(a))

    After running server.py and client.py, I get the following output:

    $ python server.py 
b'\x80\x04\x95\x17\x02\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(h\
x00\x8c\n_load_type\x94\x93\x94\x8c\x04type\x94\x85\x94R\x94\x8c\x01A\x94h\x04\x8c\x06object\x94\x85\
x94R\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x08__init__\x94h\x00\x8c\x10_creat
e_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(K\x01K\x00K\x00K\x01K\x02KCC\nd\x01|\x00
_\x00d\x00S\x00\x94NK\x02\x86\x94\x8c\x03num\x94\x85\x94\x8c\x04self\x94\x85\x94\x8c-/home/eduardo/Do
wnloads/SO_Question/client.py\x94h\x10K\rC\x02\x00\x01\x94))t\x94R\x94c__builtin__\n__main__\nh\x10NN
t\x94R\x94}\x94}\x94(\x8c\x0f__annotations__\x94}\x94\x8c\x0c__qualname__\x94\x8c\nA.__init__\x94u\x8
6\x94b\x8c\x08multiple\x94h\x12(h\x14(K\x02K\x00K\x00K\x02K\x02KCC\n|\x00j\x00|\x01\x14\x00S\x00\x94N
\x85\x94h\x18h\x19\x8c\x01n\x94\x86\x94h\x1bh(K\x10C\x02\x00\x01\x94))t\x94R\x94c__builtin__\n__main_
_\nh(NNt\x94R\x94}\x94}\x94(h#}\x94h%\x8c\nA.multiple\x94u\x86\x94b\x8c\x07__doc__\x94N\x8c\r__slotna
mes__\x94]\x94ut\x94R\x94)\x81\x94}\x94h\x17K\x02sb.'
<__main__.A object at 0x7fe8ec0656d0>
6

    Success! Thanks for the help!