Search code examples
c#unity-game-engine3dcomputer-visionaugmented-reality

Align a 3D Model over a 3D Object detect box using ZED Camera and Unity

I have a 2D object detection model using yolo, and I don't have segmentation, right now I'm using ZED stereo Camera and its SDK to get a 3D BBOX out of it, as shown

I would like to align a 3D Game object to it, the 3D BBox doesn't have orientation, so I would like to align a virtual 3D object over the real 3D Object.

I'm looking for an algorithm or an approach.

enter image description here

Solution

  • If you only have a Axis Aligned Bounding Box (AABBox) you cannot do very much. The best you can do is probably to check the size of the AABBox and match that against the AABBox of the virtual 3D object to select the closest 90-degree rotation.

    The next step up would be to somehow produce a Rotated Bounding Box . If you have that you can simply use the rotation of the detected BBox for your virtual 3D object. You will again need to check the length of the axises of the BBox make sure they align properly. 3D Oriented bounding boxes made simple seem to give a decent overview of the approach to get the rotated BBox.

    The best, but most complicated way would be to compute the relative transform between the two objects. The algorithm I'm most familiar with for this is Iterative Closest Points (ICP). Last time I looked I could not find any easy to use free implementation of ICP for .Net, and I would say that it is at least moderately challenging to implement yourself if you do not have a strong foundation in math and 3D graphics.

    Both Rotated bounding Boxes and ICP requires point clouds to operate on. For your virtual object you can just use a random set of vertices. For your image object you can use the same approach if you can somehow create a mesh. Otherwise you might be able to extract 3D points from the depth image, but that will ofc require that you have some kind of segmentation so you know what pixels to use.

    Overall I would say that this is the kind of project you find a library to do the majority of work for you. Or you should be prepared to invest a significant chunk of time and effort to write your own implementation.