How to get a depth point from screen space using Tango and Unity

Question

I want to find a point in world space at the center of the viewport using the point cloud from the Unity Tango integration. I'll then place a "reticle" type GameObject there.

I'm basing my code off the point_to_point example in the C API examples.

TangoSupport_getDepthAtPointNearestNeighbor is the function that will do what what I need, but it requires a pose parameter.

The C API says this is:

The pose of the point cloud relative to the color camera used to obtain uv_coordinates.

The Unity API says this is:

Transformation matrix of the color camera with respect to the Unity World frame.

The point_to_point example calls:

TangoSupport_calculateRelativePose(
  color_image_timestamp, TANGO_COORDINATE_FRAME_CAMERA_COLOR,
  point_cloud->timestamp, TANGO_COORDINATE_FRAME_CAMERA_DEPTH,
  &pose_color_camera_T_depth_camera);

to find the pose.

But this doesn't exist in the Unity integration. I've tried adding it manually like this:

internal const string TANGO_SUPPORT_UNITY_DLL = "tango_support_api";
[DllImport(TANGO_SUPPORT_UNITY_DLL)]
private static extern int TangoSupport_calculateRelativePose(
    double base_timestamp, TangoEnums.TangoCoordinateFrameType base_frame,
    double target_timestamp, TangoEnums.TangoCoordinateFrameType target_frame,
    out TangoPoseData base_frame_T_target_frame);

And calling it like this:

TangoPoseData poseData = new TangoPosedata();

TangoSupport_calculateRelativePose(
    pointCloudData.m_timestamp,
    TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_CAMERA_COLOR,
    pointCloudData.m_timestamp,
    TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_CAMERA_DEPTH,
    out poseData
);

But this give me a null ref when I call it. Am I taking the right approach? Does TangoSupport_calculateRelativePose exist in the tango_support_api library that ships with the Unity integration?

There is a point_to_point example in the Unity package. But uses a C# function FindClosestPoint, which just iterates through every point in the point cloud, which not a usable solution.

Answer 1

Currently, FindClosestPoint is not available through Unity. The closest substitution is the FindPlane function in PointCloud prefab. See the function here.