Convert streamed PNG data to image: data Streamed via TCP using UnrealEngine 4.10

Question

I have created a small application using UnrealEngine 4.10 (UE4). Within that application, I am grabbing the colorBuffer via ReadPixels. I am then compressing the colorBuffer to PNG. Finally, the compressed colorBuffer is sent via TCP to another application. The UE4 application is written in c++, not that that should matter. The compressed colorBuffer is being sent every "tick" of the application - essentially 30 times a second (thereabouts).

My client, the one receiving the streamed PNG is written in c# and does the following:

• Connect to server If connected
• get the stream (memory stream)
• read the memory stream into a byte array
• convert the byte array to an image

Client implementation:

private void Timer_Tick(object sender, EventArgs e)
{
   var connected = tcp.IsConnected();

    if (connected)
    {
       var stream = tcp.GetStream(); //simply returns client.GetStream();
       int BYTES_TO_READ = 16; 
       var buffer = new byte[BYTES_TO_READ];
       var totalBytesRead = 0;
       var bytesRead;
       do {
           // You have to do this in a loop because there's no            
           // guarantee that all the bytes you need will be ready when 
           // you call.
           bytesRead = stream.Read(buffer, totalBytesRead,   
                                   BYTES_TO_READ - totalBytesRead);
           totalBytesRead += bytesRead;
       } while (totalBytesRead < BYTES_TO_READ);

       Image x = byteArrayToImage(buffer);

    }
}

public Image byteArrayToImage(byte[] byteArrayIn)
{
   var converter = new ImageConverter();
   Image img = (Image)converter.ConvertFrom(byteArrayIn);
   return img;
 }

The problem is that Image img = (Image)converter.ConvertFrom(byteArrayIn);

Throws an argument exception, telling me "Parmeter is not valid".

The data being sent looks like this:

My byteArrayInand buffer look like this:

I have also tried both: Image.FromStream(stream); and Image.FromStream(new MemoryStream(bytes));

Image.FromStream(stream); causes it to read forever... and Image.FromStream(new MemoryStream(bytes)); results in the same exception as mentioned above.

Some questions:

1. What size shall I set BYTES_TO_READ to be? I set as 16 because when I check the size of the byte array being sent in the UE4 application (dataSize in the first image), it says the length is 16... Not too sure about what to set this as.

2. Is the process that I have followed correct?

What am I doing wrong?

UPDATE

@RonBeyer asked if I could verify that the data sent from the server matches that which is received. I have tried to do that and here is what I can say:

The data sent, as far as I can tell looks like this (sorry for formatting):

The data being received, looks like this:

var stream = tcp.GetStream();

int BYTES_TO_READ = 512;
var buffer = new byte[BYTES_TO_READ];

Int32 bytes = stream.Read(buffer, 0, buffer.Length);
var responseData = System.Text.Encoding.ASCII.GetString(buffer, 0, 
                                                        bytes);

//responseData looks like this (has been formatted for easier reading)
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR
//?PNG\r\n\u001a\n\0\0\0\rIHDR

If I try take a single line from the responseData and put that into an image:

  var stringdata = "?PNG\r\n\u001a\n\0\0\0\rIHDR";
  var data = System.Text.Encoding.ASCII.GetBytes(stringdata);
  var ms = new MemoryStream(data);
  Image img = Image.FromStream(ms);

data has a length of 16... the same length as the dataSize variable on the server. However, I again get the execption "Parameter is not valid".

UPDATE 2

@Darcara has helped by suggesting that what I was actually receiving was the header of the PNG file and that I needed to first send the size of the image. I have now done that and made progress:

for (TArray<class FSocket*>::TIterator ClientIt(Clients); ClientIt; 
                                                          ++ClientIt)
{
    FSocket *Client = *ClientIt;

    int32 SendSize = 2 * x * y;
    Client->SetNonBlocking(true);
    Client->SetSendBufferSize(SendSize, SendSize);
    Client->Send(data, SendSize, bytesSent);
}

With this, I am now getting the image on the first go, however, subsequent attempts fail with the same "Parameter is not valid". Upon inspection, I have noticed that the stream now appears to be missing the header... "?PNG\r\n\u001a\n\0\0\0\rIHDR". I came to this conclusion when I converted the buffer to a string using Encoding.ASCII.GetString(buffer, 0, bytes);

Any idea why the header is now only being sent to first time and never again? What can I do to fix it?

Answer 1

First of all, thank you to @Dacara and @RonBeyer for your help.

I now have a solution:

Server:

for (TArray<class FSocket*>::TIterator ClientIt(Clients); ClientIt; 
                                                          ++ClientIt)
{
    FSocket *Client = *ClientIt;

    int32 SendSize = x * y; // Image is 512 x 512 
    Client->SetSendBufferSize(SendSize, SendSize);
    Client->Send(data, SendSize, bytesSent);
}

The first issue was that the size of the image needed to be correct:

 int32 SendSize = 2 * x * y;

The line above is wrong. The image is 512 by 512 and so SendSize should be x * y where x & y are both 512.

The other issue was how I was handling the stream client side.

Client:

var connected = tcp.IsConnected();

if (connected)
{
    var stream = tcp.GetStream();

    var BYTES_TO_READ = (512 * 512)^2;
    var buffer = new byte[BYTES_TO_READ];

    var bytes = stream.Read(buffer, 0, BYTES_TO_READ);

    Image returnImage = Image.FromStream(new MemoryStream(buffer));

    //Apply the image to a picture box. Note, this is running on a separate 
    //thread.
    UpdateImageViewerBackgroundWorker.ReportProgress(0, returnImage);
 }

The var BYTES_TO_READ = (512 * 512)^2; is now the correct size.

I now have Unreal Engine 4 streaming its frames.