Microsoft Kinect V2 + Unity 3D Depth = Warping

Question

I've been working on a scene in Unity3D where I have the KinectV2 depth information coming in at 512 x 424 and I'm converting that in real time to Mesh that is also 512 x 424. So there is a 1:1 ratio of pixel data (depth) and vertices (mesh).

My end goal is to make the 'Monitor 3D View' scene found in 'Microsoft Kinect Studio v2.0' with the Depth.

I've pretty much got it working in terms of the point cloud. However, there is a large amount of warping in my Unity scene. I though it might of been down to my maths, etc.

However I noticed that its the same case for the Unity Demo kinect supplied in their Development kit.

I'm just wondering if I'm missing something obvious here? Each of my pixels (or vertices in this case) is mapped out in a 1 by 1 fashion.

I'm not sure if its because I need to process the data from the DepthFrame before rendering it to scene? Or if there's some additional step I've missed out to get the true representation of my room? Because it looks like theres a slight 'spherical' effect being added right now.

These two images are a top down shot of my room. The green line represents my walls.

The left image is the Kinect in a Unity scene, and the right is within Microsoft Kinect Studio. Ignoring the colour difference, you can see that the left (Unity) is warped, whereas the right is linear and perfect.

I know it's quite hard to make out, especially that you don't know the layout of the room I'm sat in :/ Side view too. Can you see the warping on the left? Use the green lines as a reference - these are straight in the actual room, as shown correctly on the right image.

Check out my video to get a better idea: https://www.youtube.com/watch?v=Zh2pAVQpkBM&feature=youtu.be

Code C#

Pretty simple to be honest. I'm just grabbing the depth data straight from the Kinect SDK, and placing it into a point cloud mesh on the Z axis.

//called on application start
void Start(){

    _Reader = _Sensor.DepthFrameSource.OpenReader();
    _Data = new ushort[_lengthInPixels];
    _Sensor.Open();
}

//called once per frame
void Update(){

    if(_Reader != null){

        var dep_frame = _Reader.AcquireLatestFrame();
        dep_frame.CopyFrameDataToArray(_Data);
        dep_frame.Dispose();
        dep_frame = null;

        UpdateScene();
    }
}

//update point cloud in scene
void UpdateScene(){

    for(int y = 0; y < height; y++){

        for(int x = 0; x < width; x++){

            int index = (y * width) + x;
            float depthAdjust = 0.1;
            Vector3 new_pos = new Vector3(points[index].x, points[index].y, _Data[index] * depthAdjust;
            points[index] = new_pos;
        }
    }
}

Kinect API can be found here: https://msdn.microsoft.com/en-us/library/windowspreview.kinect.depthframe.aspx

Would appreciate any advise, thanks!

Answer 1

With thanks to Edward Zhang, I figured out what I was doing wrong.

It's down to me not projecting my depth points correctly, in where I need to use the CoordinateMapper to map my DepthFrame into CameraSpace.

Currently, my code assumes an orthogonal depth instead of using a perspective depth camera. I just needed to implement this:

https://msdn.microsoft.com/en-us/library/windowspreview.kinect.coordinatemapper.aspx

//called once per frame
void Update(){

    if(_Reader != null){

        var dep_frame = _Reader.AcquireLatestFrame();
        dep_frame.CopyFrameDataToArray(_Data);
        dep_frame.Dispose();
        dep_frame = null;

        CameraSpacePoint[] _CameraSpace = new CameraSpacePoint[_Data.Length];
        _Mapper.MapDepthFrameToCameraSpace(_Data, _CameraSpace);

        UpdateScene();
    }
}

//update point cloud in scene
void UpdateScene(){

    for(int y = 0; y < height; y++){

        for(int x = 0; x < width; x++){

            int index = (y * width) + x;

            Vector3 new_pos = new Vector3(_CameraSpace[index].X, _CameraSpace[index].Y, _CameraSpace[index].Z;
            points[index] = new_pos;
        }
    }
}