Get Intel RealSense Depth Stream in Unity

Question

I'm currently trying to get the depthstream of the new RealSense Generation (D435, SDK2) as a Texture2D in Unity. I can easily access the regular RGB stream as a WebCamTexture, when I try to get the depthsream, I get this error:

Could not connect pins - RenderStream()

Unity recognizes the depthcamera, but can't display it.

I also tried to use the prefabs of the Unity Wrapper, but they don't really work for my project. If I use the prefabs, I can get the data to an R16 texture. Does anyone have an idea, how I can get the depth information at a certain point in the image (GetPixel() doesn't work for R16 textures...)? I'd prefer to get a WebCamTexture stream, if this doesn't work, I have to save the information in a different way...

Answer 1

What i did to get depth data was to create my own class inheriting from RawImage. I used my custom class as the target for the depth render stream, and got the image from the texture component in my class.

Binding to custom class

In my case i wanted to convert the 16bit depth data to an 8bit pr channel rgb pgn so that i could export it as a greyscale image. Here's how i parsed the image data:

byte[] input = texture.GetRawTextureData();
        //create array of pixels from texture 
        //remember to convert to texture2D first
    Color[] pixels = new Color[width*height];

    //converts R16 bytes to PNG32
    for(int i = 0; i < input.Length; i+=2)
    {

        //combine bytes into 16bit number
        UInt16 num = System.BitConverter.ToUInt16(input, i);
        //turn into float with range 0->1
        float greyValue = (float)num / 2048.0f;

        alpha = 1.0f;

        //makes pixels outside measuring range invisible
        if (num >= 2048 || num <= 0)
            alpha = 0.0f;

        Color grey = new Color(greyValue, greyValue, greyValue, alpha);

            //set grey value of pixel based on float
        pixels.SetValue(grey, i/2);
    }

to get the pixels you can simply access the new pixels array.