My Gstreamer 1.20 application is writing two H.264 video streams and one Opus audio stream to a MP4 file. The relevant parts of the pipeline looks as follows:
Video encoder (this part is present twice):
queue max-size-time=100000000 leaky=downstream ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux.
Audio encoder:
queue max-size-time=100000000 leaky=downstream ! opusenc bitrate-type=vbr inband-fec=true ! queue ! opusparse ! mux.
Muxer and filesink:
mp4mux name=mux fragment-duration=1000 latency=100000000 ! filesink sync=true location=out.mp4
The result is an MP4 file which I can play just fine in Totem on Ubuntu, and even switch between video streams using "Switch angles".
However, none of the following work:
In all cases, I get some messages about "Invalid NAL unit size" and/or "non-monotonous DTS". So what would I need to do to my output pipeline to get a "compliant" MP4 file?
Note: I can "fix" the resulting file by using GStreamer with qtdemux to split it into two raw H.264 files and a WAV file, and then use ffmpeg to re-encode it into MP4 with an MP3 audio track. But I'd rather have a workable MP4 file right away.
Your pipeline may lack video information from caps. You may check with:
gst-launch-1.0 \
videotestsrc ! video/x-raw,format=NV12,width=640,height=480,framerate=30/1 ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 insert-vui=1 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
videotestsrc pattern=ball ! video/x-raw,format=NV12,width=640,height=480,framerate=30/1 ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 insert-vui=1 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
audiotestsrc ! audioresample ! voaacenc ! queue ! mux. \
qtmux name=mux fragment-duration=1000 ! queue ! filesink sync=true location=out.mp4
# Or try mkv for opus:
gst-launch-1.0 \
videotestsrc ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
videotestsrc pattern=ball ! queue ! x264enc bitrate=1500 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline,stream-format=avc ! queue ! mux. \
audiotestsrc ! audioconvert ! opusenc ! queue ! mux. \
matroskamux name=mux ! queue ! filesink sync=true location=out.mkv
Once recorded, you should be able to see the streams:
# stream 0
ffplay -autoexit out.mp4
# stream 1
ffplay -autoexit out.mp4 -vst v:1