Search code examples
c#c++kinectkinect-sdk

ColorSpacePoint to DepthSpacePoint

I am trying to map a single ColorSpacePoint to a DepthSpacePoint in the Kinect SDK 2.0.

If understand the three types of coordinate systems Kinect provides, this should be a pretty easy mapping. Both ColorSpace and DepthSpace are 2 dimensional spaces they just have different resolutions and aspect ratios.

However, I cannot figure out which function in CoordinateMapper would receive a specific ColorSpacePoint and return the associated DepthSpacePoint.

Can anybody shed some light on this matter?

Thanks!

Solution

  • There is no method in the SDK that maps a single ColorSpacePoint to the DepthSpace (or to the CameraSpace). And there can't be! To get the correct mapping, you don't only need the pixel coordinates, but also the depth at that specific pixel. And in the RGB image, you don't have that information.

    To achieve a correct mapping from color space to depth or camera space, you first have to map the complete depth image to the color image, to get depth values for each color pixel. Then it would be easy map a single pixel from color to depth space. But by then you have the complete mapping anyway.

    So, if you really need to go in that direction, you need to map the whole image using CoordinateMapper.MapColorFrameToCameraSpace.

    On the other hand, when you are in the depth or camera space, you already have all the information you need at each pixel, so mapping single pixels is easy. That's why those methods exist (e.g. CoordinateMapper.MapDepthPointToCameraSpace).

    Edit:

    I hope a little drawing can help explain why the depth is needed to get a correct mapping between color and depth space:

    enter image description here

    One of the color camera's pixels is looking at the blue object (the solid blue arrow). This pixel is mapped to a specific pixel of the depth camera (solid red arrow).

    If the object wasn't there, the same pixel of the color camera would be looking at the background (dashed blue arrow), which would be mapped to a different pixel in the depth camera (dashed red arrow - different direction of the arrows means different pixels).

    But the color camera doesn't know if it's looking at the object or the background. It just knows which color is there. Not how far away that color is. That's why the color camera alone can't decide which is the correct pixel in the depth camera.

    The depth camera, on the other hand, knows the distance at each pixel, so it can figure out the corresponding pixel in the color camera.