Search code examples
c++opencvunity-game-enginedllaruco

All ArUo Tags are getting Rejected when passing the image data from Unity


Background

I am trying to detect ArUco markers from image data sent from Unity to OpenCV Functions which are accessed using their .dlls.

Versions

OpenCV : 4.5.0/4.5.1
Unity : 2020.1.17f1
Aruco dictionary : 6x6_250 family and AprilTags 36h11 family

Problem

When I use a camera and send each frame(as shown in code here), detectMarkers() function works but every tag gets rejected.

I have tried:

  • Flipping the image only in Unity using System.Array.Reverse(rawImg) * I have also tried flipping the image in c++ using flip(imageCopy, imageCopy, -1) and then back after the image is processed;
  • In a combination of both in Unity and OpenCV. But I made sure the image which was feeding into detectMarkers() function was proper(i.e not flipped) by 'imshow()`.
  • I have also changed from aruCo dictionary to April tags so that there is no version mismatch in the tags.
  • I also checked by loading the image using imread() only in .dll. This works and the tag gets detected.
  • In addition, I also tried sending single image data (using Texture2D as image type). Here the Unity crashes during the cvtColor() line.

Part of the Unity code snippet

private WebCamTexture CamTexture;

 void Start()
 {
   CamTexture = new WebCamTexture();
   CamTexture.Play();
 }

 void Update()
 {
   if (CamTexture.isPlaying)
     {
       var rawImg = CamTexture.GetPixels32();
       MarkersDetection.detectAruco(ref ArucoID, ref arrayLength, ref rawImg, WebcamWidth, WebcamHeight);
     }
}

Part of OpenCV code snippet

struct Color32
{
    uchar red;
    uchar green;
    uchar blue;
    uchar alpha;
};

extern "C" void __declspec(dllexport) __stdcall detectAruco(int& outArucoID,int& arraySize,Color32 **rawImg,int width,int height) {
cv::Mat imageCopy;
cv::Mat image(height, width, CV_8UC4, *rawImg);

cvtColor(image, imageCopy, COLOR_BGRA2BGR); 
cv::aruco::detectMarkers(imageCopy, arucoDictionary, arucoCorners, arucoIds, parameters, rejectedCandidates);
}

Solution

  • I was doing the flipping wrong! Now I flipped the image using a image tool(any editing tool which flips image works; rather than manually) and now it works.