Unity: Why is my texture flickering while receiving webcam feed over TCP?

Question

I try to receive my webcam feed in Unity over TCP and assign it as a Texture2D to my plane. It works really well but it is flickering, so my guess is that the texture is loading in empty textures because it is updating faster then the TCP client receiving it.

Here is my python sending code:

import socket
import cv2, imutils
from argparse import ArgumentParser


parser = ArgumentParser()
parser.add_argument("--connection", default="127.0.0.1", type=str, help="ip to connect to")
args = parser.parse_args()
host, port = "127.0.0.1", 25001
host = args.connection
data = "1,2,3"

# SOCK_STREAM means TCP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

vid = cv2.VideoCapture(0) #  replace 'rocket.mp4' with 0 for webcam
fps,st,frames_to_count,cnt = (0,0,20,0)
try:
    # Connect to the server and send the data
    sock.connect((host, port))
    print('waiting for a connection', flush=True)

    print("connected, trying to send data...", flush=True)
    WIDTH=400
    while(vid.isOpened()):
        _,frame = vid.read()
        frame = imutils.resize(frame,width=WIDTH)
        encoded,buffer = cv2.imencode('.jpg',frame,[cv2.IMWRITE_JPEG_QUALITY,80])
        #message = base64.b64encode(buffer)
        #print(sys.getsizeof(buffer))
        sock.sendall(bytes(buffer))
        print("Data sent, response: ", flush=True)
        response = sock.recv(65536)

        frame = cv2.putText(frame,'FPS: '+str(fps),(10,40),cv2.FONT_HERSHEY_SIMPLEX,0.7,(0,0,255),2)
        frame = cv2.putText(frame,'ip: '+str(host),(10,80),cv2.FONT_HERSHEY_SIMPLEX,0.7,(0,0,255),2)
        cv2.imshow('TRANSMITTING VIDEO',frame)
        key = cv2.waitKey(1) & 0xFF
        if key == ord('q'):
            sock.close()
            break
        if cnt == frames_to_count:
            try:
                fps = round(frames_to_count/(time.time()-st))
                st=time.time()
                cnt=0
            except:
                pass
        cnt+=1

finally:
    sock.close()

And here my receiver in C#:

public class ReceiveImages : MonoBehaviour
{
Thread thread;
    public int connectionPort = 25001;
    TcpListener server;
    TcpClient client;
    bool running;
    private Texture2D mainTexture;
    private byte[] recvBuffer;
    public GameObject renderImage;


    void Start()
    {
        Application.targetFrameRate = 40;
        // Receive on a separate thread so Unity doesn't freeze waiting for data
        ThreadStart ts = new ThreadStart(GetData);
        thread = new Thread(ts);
        thread.Start();
        mainTexture = new Texture2D(2, 2);
    }

    void GetData()
    {
        // Create the server
        server = new TcpListener(IPAddress.Any, connectionPort);
        server.Start();

        // Create a client to get the data stream
        client = server.AcceptTcpClient();

        // Start listening
        running = true;
        while (running)
        {
            Connection();
        }
        server.Stop();
    }

    void Connection()
    {
        // Read data from the network stream
        NetworkStream nwStream = client.GetStream();
        byte[] buffer = new byte[client.ReceiveBufferSize];
        int bytesRead = nwStream.Read(buffer, 0, client.ReceiveBufferSize);
        
        string dataReceived = Encoding.UTF8.GetString(buffer, 0, bytesRead);
        
        if (dataReceived != null)
        {
            recvBuffer = buffer;
            nwStream.Write(buffer, 0, bytesRead);
        }
    }

    // Use-case specific function, need to re-write this to interpret whatever data is being sent
    public static void ParseData(byte[] textureData, Texture2D destination)
    {
        destination.LoadImage(textureData);
    }

    void Update()
    {
        // Set this object's position in the scene according to the position received
        //transform.position = position;
        ParseData(recvBuffer, mainTexture);
        renderImage.GetComponent<Renderer>().material.mainTexture = mainTexture;

    }
}

As I said I assume that it has something to do with receiving the buffer, but how can I check for the size of the estimated package?

I appreciate all the help,

Best wishes!

Answer 1

There is no guarantee that you you will read the entire frame in a single nwStream.Read call, or that all read bytes belong to the same frame. You need some form of message frameing to ensure that each frame is received in its entirety.

The simplest form of message framing for a image is to just write the buffer size before the actual buffer. And on the read side read the buffer size, and then read in a loop until all bytes for the image has been received.

Or, just use a protocol that takes care of this for you. Something like HTTP, gRCP or MQTT will let you send messages instead of just having a stream, and should be available in both pyton and C#.

You also need to be careful with thread safety. Your example updates shared data without any form of synchronization. This might be ok if you are just setting a reference, but I would be much more happy with a lock, since various levels of optimization may cause unexpected things to happen if proper synchronization is not used.