Gstreamer Playing a raw h.264 stream from a USB source (timestamp issues)

I'm dealing with a live video source (drone wireless video receiver) that outputs a raw h.264 video stream over USB. My goal is to integrate it into QGroundStation in Android, which has a GStreamer pipeline.

I have dumped a slice of the received USB data to a file, which is perfectly playable with vlc using the following command:

vlc -c dump.bin --demux h264

However, if I play it back using this GStreamer pipeline, the playback speed is too high (like x10)

gst-launch-1.0.exe filesrc location="dump.bin" ! h264parse ! avdec_h264 ! autovideosink

I'm using appsrc to push the USB data into the QGroundControl pipeline. The video plays, but lots of frames are dropped and gstreamer complains about packets dropped because frames are too late.

[...] 
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.404134207)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.403025291)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.401385832)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.400435290)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.399607540)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398911040)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.398131998)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.397308623)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.396620290)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395761040)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.395125498)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.394197123)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.393461831)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.392803831)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391983373)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.391033998)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.389664914)
    W QGroundControl Daily: GStreamerAPILog: <amcvideodec-omxrkvideodecoderavc18> Frame is too late, dropping (deadline -0:00:00.388862831)
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 38639 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 37460 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 46566 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 36055 will be dropped
    W QGroundControl Daily: GStreamerAPILog: <h264parse53> broken/invalid nal Type: 1 Slice, Size: 43397 will be dropped
[...]

After closer inspection of my dump, I realized that the stream is lacking pts and dts information (which seems to be usual in baseline h.264 streams)

ffprobe -show_frames dump.bin

[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=1843
pos=0
flags=K_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=48000
duration_time=0.040000
convergence_duration=N/A
convergence_duration_time=N/A
size=16851
pos=1843
flags=K_
[/PACKET]

But apparently, the duration information is there.

The USB endpoint reads 512-byte chunks (due to the USB Hi-Speed max. payload size for a bulk endpoint), and some transfers are smaller (400+ bytes long). I have no way to detect the beginning/end of NALs since it's an opaque continuous byte stream. (video/x-h264, stream-format=(string)byte-stream, alignment=none)

So I built an appsrc to push the video stream to the pipeline and tried to blindly timestamp the buffers like this:

void _startFeed(GstElement *source, guint size, gpointer pContext) {
(void)pContext;

GstBuffer *pBuffer;
GstFlowReturn ret;
GstMapInfo map;
guint8 *pRaw;

GstClock *pClk = gst_element_get_clock(gPipeline);
GstClockTime absolute = gst_clock_get_time(pClk);
GstClockTime base = gst_element_get_base_time(gPipeline);
gst_object_unref(pClk);

pBuffer = gst_buffer_new_and_alloc(bufferSize);

// Timestamp the buffers
GST_BUFFER_PTS(pBuffer) = absolute - base;
GST_BUFFER_DTS(pBuffer) = GST_BUFFER_PTS(pBuffer);

gst_buffer_map(pBuffer, &map, GST_MAP_WRITE);

pRaw = (guint8 *)map.data;

fifo_pull(pRaw, size);  // This pulls up to 'size' bytes from the USB FIFO and copies it to pRaw

gst_buffer_unmap(pBuffer, &map);

g_signal_emit_by_name(gAppSrc, "push-buffer", pBuffer, &ret);

gst_buffer_unref(pBuffer);
}

... but still no luck ...

I have had limited success by using the following pipeline that encodes the h.264 stream into RTP payloads and then decodes it with a caps filter specifying the target framerate:

gst-launch-1.0.exe filesrc location="dump.bin" ! video/x-h264, stream-format=(string)byte-stream, alignment=none ! h264parse ! rtph264pay ! rtpjitterbuffer ! rtph264depay ! video/x-h264, stream-format=byte-stream, alignment=nal, framerate=(fraction)30/1 ! h264parse ! avdec_h264 ! autovideosink

I could build that into QGroundControl in C++ but I don't think it's the right approach and I should not make any assumptions about the target framerate since in this case it's 30 fps, but it may change dynamically.

So, my questions are:

What would be the right approach to getting the video playing at the right speed without any frame drops?
Is it reasonable or possible to ask GStreamer to generate the PTS/DTS (there are no B-frames, so PTS should be equal to DTS) based on the duration information of the packets using the standard pipeline?

Update::

I have tried a known workaround for interpolating PTS from DTS + duration described here but in my case, neither PTS nor DTS are available in the stream, so it didn't make any difference.
The correct framerate for this particular video dump should be 29.97 fps (NTSC). I have a running counter in the video image to verify it.
As you can see in my ffprobe output, the duration_time information appears as 0.04 s, which corresponds to 25 fps, which puzzles me.
Surprisingly, VLC plays the video at the right speed, and correctly guesses the framerate as 29.97 fps (as per the video codec info window that you get pressing Ctrl+J)
I've noticed that rtph264pay will generate the missing timestamps, but the guessed framerate is 25 fps, as the duration_time field of the packets suggests

Solution

I used an h.264 stream analyzer and realized that this particular stream has no VUI information in the SPS NALs, (therefore no time_tick or time_scale info that could be used by the parser to calculate PTS).

What worked for me is to set the property "sync" = false on the sink to render the frames as soon as they arrive. Not ideal, but I can live with it.