Search code examples
androidandroid-cameragoogle-project-tangoandroid-camera2artoolkit

Adding ARToolkit Marker tracking into Tango

I have been trying to integrate ARToolkit Marker Object tracking into a Tango Application.

So far I have created a build so that a tango app can access and use the ARToolkit Native Library or the ARToolkit Unity wrappers. However, they both seem to require exclusive access to the camera in their default configurations.

How could you feed the same Android video feed to both libraries?

  • Could you create a dummy camera device which doubles out the feed?
  • Could you take the tango feed as normal, and then resend it into ARToolkit with a special VideoConf

[edit]
ARToolkit uses the older Camera1 API, takes a onPreviewFrame() callback and passes that byte[] data to it's own Native Library call, which does the actual work.

Along the lines of the second bullet point, could Tango provide a copy of each frames raw camera data using something like iTangoVideoOverlay .
(ARToolkits NDK functionality seems to expect NV21, but can also accept other formats)

If that data was extractable from tango, I believe the ARToolkit NDK functionality can be used without actually owning the camera.

Solution

  • This blog shows an example integrating the two.

    It also links to example source code, but I haven't tidied it up at all after testing - proceed with caution!

    You cannot feed the same camera source to both libraries (first bullet point), but you can forward the camera feed from Tango (ITangoVideoOverlay) into ARToolkit ([AcceptVideoImage][2]) (second bullet point).
    This is not ideal, because it is fairly inefficient to send the data to Java from C#. The Phab 2 Pro has to downsample the video X4 to achieve a decent framerate.
    A better answer would replace the AndroidJavaClass calls with pipes/sockets. Also there are many little problems - it's a pretty hacky workaround.