How can I send PNG image over TCP sockets using ZeroMQ?

Question

I have a Keras model working on Python and I want to send frames to this model from Unity 3D camera. I can easily transport strings among them by using an external library. This external library sends a byte array to Python. So, I converted Unity's camera frames to a byte array. However, I do not know how to read the sent images (byte array) on Python.

In order to handle this problem, I have tried some Sender-Receiver codes using only Python. However, those did not work.

Here are the codes:

Sender :

import zmq
import base64


context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")

f = open("sample.png", 'rb')
bytes = bytearray(f.read())
strng = base64.b64encode(bytes)
socket.send(strng)
f.close()

Receiver :

import zmq
import base64


context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

import base64
message = socket.recv()
f = open("sample.png", 'wb')
ba = bytearray(base64.b64decode(message))
f.write(ba)
f.close()

Is that a good way for sending frames from Unity to Python? Or are there other ways? How can I handle my problem? What is wrong with the codes above?

Answer 1

Q : "Is that a good way for sending frames from Unity to Python?"

I dare to say.

_{However, I do not know how to read the sent images (byte array) on Python.}

Why would ever one use a tool to "encode" something,
without having the matching tool to "decode"?

---

Q : "Or are there other ways?"

Oh sure there are.

---

Q : "How can I handle my problem?"

Best learn such tools, that you can safely use, both for the encode & the decode phases. Plenty of such available as Python standard modules - numpy with pil/pillow or cv for image conversions, dill or pickle for SER/DES or even a struct.

Having used these for ages with ZeroMQ, you may be sure to go in the right direction using safe tools.

---

Q : "What is wrong with the codes above?"

This will never fly.

The "Sender",
having been proposed as a REP-instance can never start with a call to the .send()-method.
Never.

The "Receiver",
having been proposed as a REQ-instance can never start with a call to the .recv()-method.
Never.

Sounds complicated? Yes, the choice of REQ/REP was not mine.

ZeroMQ REQ/REP Formal Scalable Communication Archetype is one of the most complicated ( operating an internal distributed Finite-State-Automaton (dFSA), actually 1:N of them, having not a risk, but a solid certainty of un-salvageably self-deadlocking any one of these N dFSA and the story ends... )

Do not get me wrong, I love ZeroMQ since ever and admire Pieter HINTJENS' & Martin SUSTRIK's immense amounts of invention & performance polishing they've breathed into the ZeroMQ's Zen of Zero, yet I try you to warn it takes some time to master the Zen start to end.

The link above has a full solution of using rather the PUB/SUB for image-sending plus some performance tweaking hints for further thinking.