C# Unity - Extract pixel array from camera in unity and display in python

Question

I am trying to do two things:

1. Retrieve the 84 by 84 by 3 pixel array from a camera
2. Send this array to a python script where it can be displayed

I am stuck with 1), so for the moment I am simply writing the bytearray to a txt file and then opening the file in python. However, I keep getting this image which leads me to believe the bytearray is not actually being instantiated correctly with the pixel data. [] 1

Here is the C# unity code:

using System.IO;
using UnityEngine;
using MLAgents.Sensors;
// using UnityEngine.CoreModule;

public class TrainingAgent : Agent, IPrefab
{

    public Camera cam;
    private RenderTexture renderTexture;
    public int bytesPerPixel;
    private byte[] rawByteData;
    private Texture2D texture2D;
    private Rect rect;


    public override void Initialize()
    {

        renderTexture = new RenderTexture(84, 84, 24);
        rawByteData = new byte[84 * 84 * bytesPerPixel];
        texture2D = new Texture2D(84, 84, TextureFormat.RGB24, false);
        rect = new Rect(0, 0, 84, 84);
        cam.targetTexture = renderTexture;
    }

    public override void CollectObservations(VectorSensor sensor)
    {
        run_cmd();
    }

    private void run_cmd()
    {

        cam.Render();
        // Read pixels to texture
        RenderTexture.active = renderTexture;
        texture2D.ReadPixels(rect, 0, 0);
        Array.Copy(texture2D.GetRawTextureData(), rawByteData, rawByteData.Length);
        File.WriteAllBytes("/Path/to/byte/array/Foo.txt", rawByteData); // Requires System.IO
    }
}

And here is the python code:

from PIL import Image
import numpy as np
fh = open('/Path/to/byte/array/foo.txt', 'rb')
ba = bytearray(fh.read())
data = np.array(list(ba)).reshape(84,84,3)
img = Image.fromarray(data, 'RGB')
img.show()

Any help would be much appreciated as I have no idea where I am going wrong and my debugging attemps seem futile.

Answer 1

I'm not sure (don't know python in detail) but I don't think GetRawTextureData is what you would want to use here.

You rather would export an actual image binary file e.g. a JPG using ImageConversion.EncodeToJPG (former Texture2D.EncodeToJPG)

Encodes this texture into JPG format.

The returned byte array is the JPG "file". You can write them to disk to get the JPG file, send them over the network, etc.

This function works only on uncompressed, non-HDR texture formats. You must enable the texture's Read/Write Enabled flag in the Texture Import Settings. The encoded JPG data will have no alpha channel.

And then load it in python as an actual image file.

In your code this would be equivalent to replacing the last two C# lines with:

rawByteData = ImageConversion.EncodeToJPG(texture2D);
File.WriteAllBytes("/Path/to/jpg/foo.jpg", rawByteData);