Creating OpenCv mat from Gstreamer creates grey image, but I want color

Question

I use gstreamer to access webcam images. I use an appsrc to gain access to these images, and then I want to be able to process them with openCv. So first I need to load them into a Mat object. However, I am only able to do this in grey scale. If I try to read multiple channels I get a 'read access violation' exception.

The code that I use to create the Mat is as follows:

GstSample* sample;
    GstBuffer* buffer;
    GstMapInfo map;

    g_signal_emit_by_name(sink, "pull-sample", &sample);
    g_print("Check frame");
    if (sample != NULL) {

        buffer = gst_sample_get_buffer(sample);
        if (gst_buffer_map(buffer, &map, GST_MAP_READ))
        {


            Mat frame(Size(width, height), CV_8UC3, map.data, cv::Mat::AUTO_STEP);
            imwrite("elephant.jpg", frame);


        }
        g_print("Found frame");
        return GST_FLOW_OK;
    }
    return GST_FLOW_ERROR;

When I used a filesink before to write to file the images were in color.

I use the following filtercaps in my code:

filtercaps = gst_caps_new_simple("image/jpeg", "format", G_TYPE_STRING, "RGB", "width", G_TYPE_INT, width,
        "height", G_TYPE_INT,
        height, "framerate", GST_TYPE_FRACTION, 30,
        1, NULL);

I've tried putting this filter both just after the source and just before the sink, neither solved the problem. I also have a jpeg decoder in my pipeline.

I truly have no idea how to fix this. Any help or tips are greatly appreciated!

Answer 1

I figured it out on my own. The problem was that the output of the pipeline wasn't in RGB. I tried implementing a capsfilter to solve this problem, but this wasn't working out (I don't understand why).

I solved it by simply converting the output of the pipeline. In here you can see that the preferred output is I420 (A YUV color format). I used the code suggested in this post to transform the BGRA. My final code looked like this:

GstSample* sample;
    GstBuffer* buffer;
    GstMapInfo map;
    GstMemory *mem;
    mem = gst_allocator_alloc(NULL, 1000000, NULL);
    gst_memory_map(mem, &map, GST_MAP_WRITE);
    g_signal_emit_by_name(sink, "pull-sample", &sample);
    g_print("Check frame");
    if (sample != NULL) {

        buffer = gst_sample_get_buffer(sample);
        if (gst_buffer_map(buffer, &map, GST_MAP_READ)) 
        {

            g_print("size: %d", map.size);
            Mat frameYUV(height+height/2, width, CV_8UC1 , map.data, cv::Mat::AUTO_STEP);
            cv::Mat frameRGB(height, width, CV_8UC4);
            cv::cvtColor(frameYUV, frameRGB, CV_YUV2BGRA_I420);
            imwrite("elephant.jpg", frameRGB);


        }
        g_print("Found frame");
        return GST_FLOW_OK;
    }
    gst_memory_unmap(mem, &map);
    return GST_FLOW_ERROR;