Rendering issue on Project Tango using OpenCV image processing

Question

I came across one problem to render the camera image after some process on its YUV buffer.

I am using the example video-overlay-jni-example and in the method OnFrameAvailable I am creating a new frame buffer using the cv::Mat...

Here is how I create a new frame buffer:

cv::Mat frame((int) yuv_height_ + (int) (yuv_height_ / 2), (int) yuv_width_, CV_8UC1, (uchar *) yuv_temp_buffer_.data());

After process, I copy the frame.data to the yuv_temp_buffer_ in order to render it on the texture: memcpy(&yuv_temp_buffer_[0], frame.data, yuv_size_);

And this works fine...

The problem starts when I try to execute an OpenCV method findChessboardCorners... using the frame that I've created before.

The method findChessboardCorners takes about 90ms to execute (11 fps), however, it seems to be rendering in a slower rate. (It appears to be rendering in ~0.5 fps on the screen).

Here is the code of the OnFrameAvailable method:

void AugmentedRealityApp::OnFrameAvailable(const TangoImageBuffer* buffer) {

    if (yuv_drawable_ == NULL){
        return;
    }

    if (yuv_drawable_->GetTextureId() == 0) {
        LOGE("AugmentedRealityApp::yuv texture id not valid");
        return;
    }

    if (buffer->format != TANGO_HAL_PIXEL_FORMAT_YCrCb_420_SP) {
        LOGE("AugmentedRealityApp::yuv texture format is not supported by this app");
        return;
    }

    // The memory needs to be allocated after we get the first frame because we
    // need to know the size of the image.
    if (!is_yuv_texture_available_) {
        yuv_width_ = buffer->width;
        yuv_height_ = buffer->height;
        uv_buffer_offset_ = yuv_width_ * yuv_height_;

        yuv_size_ = yuv_width_ * yuv_height_ + yuv_width_ * yuv_height_ / 2;

        // Reserve and resize the buffer size for RGB and YUV data.
        yuv_buffer_.resize(yuv_size_);
        yuv_temp_buffer_.resize(yuv_size_);
        rgb_buffer_.resize(yuv_width_ * yuv_height_ * 3);

        AllocateTexture(yuv_drawable_->GetTextureId(), yuv_width_, yuv_height_);
        is_yuv_texture_available_ = true;
    }

    std::lock_guard<std::mutex> lock(yuv_buffer_mutex_);
    memcpy(&yuv_temp_buffer_[0], buffer->data, yuv_size_);

    ///
    cv::Mat frame((int) yuv_height_ + (int) (yuv_height_ / 2), (int) yuv_width_, CV_8UC1, (uchar *) yuv_temp_buffer_.data());

    if (!stam.isCalibrated()) {
        Profiler profiler;
        profiler.startSampling();
        stam.initFromChessboard(frame, cv::Size(9, 6), 100);
        profiler.endSampling();
        profiler.print("initFromChessboard", -1);
    }
    ///

    memcpy(&yuv_temp_buffer_[0], frame.data, yuv_size_);
    swap_buffer_signal_ = true;
}

Here is the code of the method initFromChessBoard:

bool STAM::initFromChessboard(const cv::Mat& image, const cv::Size& chessBoardSize, int squareSize)
{
    cv::Mat rvec = cv::Mat(cv::Size(3, 1), CV_64F);
    cv::Mat tvec = cv::Mat(cv::Size(3, 1), CV_64F);

    std::vector<cv::Point2d> imagePoints, imageBoardPoints;
    std::vector<cv::Point3d> boardPoints;

    for (int i = 0; i < chessBoardSize.height; i++)
    {
        for (int j = 0; j < chessBoardSize.width; j++)
        {
            boardPoints.push_back(cv::Point3d(j*squareSize, i*squareSize, 0.0));
        }
    }

    //getting only the Y channel (many of the functions like face detect and align only needs the grayscale image)
    cv::Mat gray(image.rows, image.cols, CV_8UC1);
    gray.data = image.data;

    bool found = findChessboardCorners(gray, chessBoardSize, imagePoints, cv::CALIB_CB_FAST_CHECK);

#ifdef WINDOWS_VS
    printf("Number of chessboard points: %d\n", imagePoints.size());
#elif ANDROID
    LOGE("Number of chessboard points: %d", imagePoints.size());
#endif

    for (int i = 0; i < imagePoints.size(); i++) {
        cv::circle(image, imagePoints[i], 6, cv::Scalar(149, 43, 0), -1);
    }
}

Is anyone having the same problem after process something in the YUV buffer to render on the texture?

I did a test using other device rather than the project Tango using camera2 API, and the rendering process on the screen appears to be the same rate of the OpenCV function process itself.

I appreciate any help.

Answer 1

I had a similar problem. My app slowed down after using the copied yuv buffer and doing some image processing with OpenCV. I would recommand you to use the tango_support library to access the yuv image buffer by doing the following:

In your config function:

int AugmentedRealityApp::TangoSetupConfig() {
   TangoSupport_createImageBufferManager(TANGO_HAL_PIXEL_FORMAT_YCrCb_420_SP, 1280, 720, &yuv_manager_);
}

In your callback function:

void AugmentedRealityApp::OnFrameAvailable(const TangoImageBuffer* buffer) {
   TangoSupport_updateImageBuffer(yuv_manager_, buffer);
}

In your render thread:

void AugmentedRealityApp::Render() {
   TangoImageBuffer* yuv = new TangoImageBuffer();
   TangoSupport_getLatestImageBuffer(yuv_manager_, &yuv);
   cv::Mat yuv_frame, rgb_img, gray_img;
   yuv_frame.create(720*3/2, 1280, CV_8UC1);
   memcpy(yuv_frame.data, yuv->data, 720*3/2*1280);  // yuv image
   cv::cvtColor(yuv_frame, rgb_img, CV_YUV2RGB_NV21); // rgb image
   cvtColor(rgb_img, gray_img, CV_RGB2GRAY); // gray image
}

You can share the yuv_manger with other objects/threads so you can access the yuv image buffer wherever you want.