Gstreamer Appsink not getting Data from the Pipeline

Question

I am designing a pipeline to Encode a video frame from a opencv application (got from a web cam) to video/x-h264 format, send it via network and decode it on another device of different type (probably a raspberry pi ) to a proper RGB stream for my project.

For this I am supposed to use a hardware accelerated Encoder and Decoder. Since , the whole scenario is huge , the current development is performed on a Intel machine using the gstreamer VAAPI plugins(vaapiencode_h264 & vaapidecode ) . Ánd also, the fact that we need to NOT use any of the networking plugins like TCPServer or UDPServer

For this I have used the below pipeline for my purpose : On the Encoder End:

appsrc name=applicationSource ! videoconvert ! video/x-raw, format=I420, width=640, height=480,framerate=30/1, pixel-aspect-ratio=1/1,interlace-mode=progressive ! vaapiencode_h264 bitrate=600 tune=high-compression ! h264parse config-interval=1 ! appsink name=applicationSink sync=false

The Appsrc part works perfectly well while the appsink part is having some issue with it.

The appsink part of this pipeline has been set with the below caps:

"video/x-h264, format=(string){avc,avc3,byte-stream },alignment=(string){au,nal};video/mpeg, mpegversion=(int)2, profile=(string)simple"

The code for the data extraction of my appsink is

    bool HWEncoder::grabData()
{

    // initial checks..

    if (!cameraPipeline)
    {

        GST_ERROR("ERROR AS TO NO PIPE FOUND ... Stopping FRAME GRAB HERE !! ");
        return false;
    }


    if (gst_app_sink_is_eos (GST_APP_SINK(applicationSink)))
    {

        GST_WARNING("APP SINK GAVE US AN EOS! BAILING OUT ");
        return false;
    }

    if (sample)
    {
        cout << "sample available ... unrefing it ! "<< endl;
        gst_sample_unref(sample);
    }


    sample = gst_app_sink_pull_sample (GST_APP_SINK(applicationSink));

    if (!sample)
    {
        GST_WARNING("No valid sample");
        return false; // no valid sample pulled !
    }

    sink_buffer = gst_sample_get_buffer(sample);

    if (!sink_buffer)
    {
        GST_ERROR("No Valid Buffer ");return false;
    }

    return true;
}

After bringing up the pipeline and checking for the buffer filling up in my appsink, I am getting stuck at the below said lines ofmy code indefinitely:

sample = gst_app_sink_pull_sample (GST_APP_SINK(applicationSink));

I have the following questions : 1) Is my Caps for appsink correct ? If not How can I determine the caps for them ? 2) Is there something wrong in my pipeline above ?

How can I fix this issue with Appsink ??

Any kind of help would be useful!

Thanks !!

Answer 1

Just a guess (I had similar problems) the problem having appsink and appsrc in same pipeline may be that when you fill/empty one of them it will block the other(more on that below).

appsink and appsrc would block when they are full/empty - this is normal desired behaviour. There is option drop for appsink or for appsrc there is option block - but using these it may be just workaround and you will get glitches in your stream. Proper solution is to handle the synchronisation between appsrc and appsink in a better way.

You can react on appsrc signals enough-data and need-data - this is our way. Also we fiddled with properties of appsrc: is-live, do-timestamp and buffer size (this may or may not help you):

g_object_set(src->appsrc,
    "stream-type", GST_APP_STREAM_TYPE_STREAM,
    "format", GST_FORMAT_TIME,
    "do-timestamp", TRUE,
    "is-live", TRUE,
    "block", TRUE,
    NULL);

Why do they block each other? Because (I guess) you process appsink and at the same time appsrc in main application thread. When one of the appsink/appsrc block the thread there is no one that would handle the processing for the other one. So when appsink is blocked because it does not have any data there is noone that can feed appsrc with new data - thus endless deadlock.

We also implemented noblock version of appsink *pull_sample method but it was just a workaround and resulted in more problems than solutions.

If you want to debug what is happening you can add GST_DEBUG entry for appsrc/appsink (I do not remember what they were), you can add callback on mentioned enough-data and need-data signals or you may add queues and enable GST_DEBUG=queue_dataflow:5 to see which queue is filled first etc.. this is always helpful when debugging the "data-deadlock".