OpenCV-OpenGL interoperability

Question

I'm under Linux and compiled OpenCV 2.4.4 with OpenGL support, but I don't have any idea of how using the opengl_interop.hpp functions (some of them are even undocumented!, at least on my version of documentation). Looking at window.cpp in the section with OpenGL enabled I found some hints about the use of the functions setOpenGLContext, setOpenGLDrawCallback and updateView but I can't get working even this very simple piece of code:

#include <opencv2/opencv.hpp>
#include <GL/gl.h>
#include <GL/glut.h>
#include <opencv2/core/opengl_interop.hpp>

using namespace cv;

void on_opengl(void* userdata);

int main(void)
{
    VideoCapture webcam(CV_CAP_ANY);
    Mat frame;
    namedWindow("window", CV_WINDOW_OPENGL);
    setOpenGlContext("window");
    while(waitKey(30) < 0)
    {
        webcam >> frame;
        setOpenGlDrawCallback("window", on_opengl);
        imshow("window", frame);
        updateWindow("window");
    }

    return 0;
}

void on_opengl(void* userdata)
{
    glLoadIdentity();

    glTranslated(0.0, 0.0, 1.0);

    glRotatef( 55, 1, 0, 0 );
    glRotatef( 45, 0, 1, 0 );
    glRotatef( 0, 0, 0, 1 );

    static const int coords[6][4][3] = {
        { { +1, -1, -1 }, { -1, -1, -1 }, { -1, +1, -1 }, { +1, +1, -1 } },
        { { +1, +1, -1 }, { -1, +1, -1 }, { -1, +1, +1 }, { +1, +1, +1 } },
        { { +1, -1, +1 }, { +1, -1, -1 }, { +1, +1, -1 }, { +1, +1, +1 } },
        { { -1, -1, -1 }, { -1, -1, +1 }, { -1, +1, +1 }, { -1, +1, -1 } },
        { { +1, -1, +1 }, { -1, -1, +1 }, { -1, -1, -1 }, { +1, -1, -1 } },
        { { -1, -1, +1 }, { +1, -1, +1 }, { +1, +1, +1 }, { -1, +1, +1 } }
    };

    for (int i = 0; i < 6; ++i) {
                glColor3ub( i*20, 100+i*10, i*42 );
                glBegin(GL_QUADS);
                for (int j = 0; j < 4; ++j) {
                        glVertex3d(0.2*coords[i][j][0], 0.2 * coords[i][j][1], 0.2*coords[i][j][2]);
                }
                glEnd();
    }
}

What is the right way of using opengl on the webcam stream?

Answer 1

OpenGL is designed for rendering graphics, OpenCV is for computer vision. Thus I suggest you using CV in a GL-based application, instead of using CV API for rendering, callback etc.

If all you want is a simple demo, then you can use freeGLUT to write a very simple program with a few callbacks, freeGLUT will handle window callbacks and GL context creation. ( GLFW or Qt is also OK ) Within the program, use the cv::ogl::Texture2D class to handle texture objects. Use Texture2D::copyFrom(...) and Texture2D::copyTo(...) to handle device/host memory transfer. Inside render callback, use standard GL routine to draw a fullscreen rect. This method is not efficient though, it works.

#include <GL/glew.h>
#include <GL/freeglut.h>
#include <opencv2/opencv.hpp>
#include <opencv2/core/opengl_interop.hpp>

using namespace cv;

//Global vars
Texture2D g_img;
void timer_cb( int )
{
    //...Update the content of g_img
}

void resize_cb( int w, int h ) { /*...*/ }
void render_cb() {
    /* ...render g_img here */
    g_img.bind();
#ifdef USE_FIXED_PIPELINE
//use fixed pipeline for old-school GL rendering
    glMatrixMode( GL_MODELVIEW );
    //Do some transformation
    glBegin(GL_QUADS);
        glTexCoord(...);
        glVertex**(...);
        ...
    glEnd();
#else
//use shaders and VBOs for 3.1+ GL
    glBindProgram( ... );
    glBindBuffer( ... );
    glVertexAttribPointer( ... );
    glDrawArrays( ... );
#endif
}
int main( int argc, char** argv )
{
    //...init GLUT, GLEW and other stuff
    glutMainLoop();
    return 0;
}

Note:

1. freeGLUT is recommanded instead of GLUT, they are TWO things. GLUT is outdated.But freeGLUT keeps supporting latest OpenGL version while expanding GLUT.
2. You will probably need a GL loading library like GLEW to get GL function pointers
3. Newer OpenGL (3.1+) no longer supports fixed pipeline rendering, therefore requires VBOs and shaders. If you target a lower version of GL, you will need to specify context version. This can be done via glutInitContextVersion( int major, int minor ). Lots of tutorials available on the web.