Search code examples
gstreamerrtmpnvidia-jetsonnvidia-jetson-nano

GStreamer: Multiple RTMP sources, Picture in Picture to mux on a Jetson Nano, then to be used with RTMP pipeline with Belabox

My goal is to have (2) RTMP sources in a Picture in Picture composition, encoding it into h265 mpegts, muxing audio from only the cam1 rtmp source, then sending it to the appsink;

This is how I see it in my mind, but I'm probably wrong: How I see it in my mind

[Confirmed] Working (On Device) Picture in Picture Pipeline:

Devices used: Camlink 4k (Sony Action Cam FDR-x3000) and Logitech c920

v4l2src device=/dev/video0 ! nvvidconv ! queue ! comp.sink_0
v4l2src device=/dev/video1 ! video/x-raw, width=800, height=448, framerate=30/1, format=YUY2 !
videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! queue ! comp.sink_1
 
nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=1266 sink_1::ypos=706 !
queue ! identity name=v_delay signal-handoffs=TRUE ! nvvidconv interpolation-method=5 !
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps !
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 !
mpegtsmux name=mux ! appsink name=appsink
 
alsasrc device=hw:2 ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 !
audioconvert ! opusenc bitrate=320000 ! opusparse ! queue ! mux.

[Confirmed] Working RTMP Pipeline:

Device used: Samsung s10e using Larix Broadcaster to stream x264 via RTMP

rtmpsrc location=rtmp://127.0.0.1/live/cam1 ! 
flvdemux name=demux 

demux.video ! identity name=v_delay signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! 
textoverlay text='' valignment=top halignment=right font-desc="Sans, 10" name=overlay ! queue ! 
videorate ! video/x-raw,framerate=60/1 !
nvvidconv interpolation-method=5 ! 
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. 

demux.audio ! aacparse ! avdec_aac ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! 
audioconvert ! opusenc bitrate=128000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux. 

mpegtsmux name=mux ! 
appsink name=appsink

All my attempts have failed;

These are My Attempts:

Attempt 1:

rtmpsrc name=cam1 location=rtmp://127.0.0.1/live/cam1 ! flvdemux name=demux0 ! queue ! demux0.video ! identity name=v_delay signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! queue ! comp.sink_0 
rtmpsrc name=cam2 location=rtmp://127.0.0.1/live/cam2 ! flvdemux name=demux1 ! queue ! demux1.video ! identity signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! queue ! comp.sink_1 

nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=0 sink_1::ypos=240 sink_1::width=320 sink_1::height=240 ! 
videorate ! video/x-raw,framerate=60/1 ! 
nvvidconv interpolation-method=5 ! 

nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. 

demux0. ! queue ! audio/mpeg ! decodebin ! audioconvert ! audioresample ! autoaudiosink 

mpegtsmux name=mux ! 
appsink name=appsink

Attempt 2:

rtmpsrc name=cam1 location=rtmp://127.0.0.1/live/cam1 ! 
flvdemux name=demux0
demux0.video ! identity name=v_delay0 signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! queue ! comp.sink_0

rtmpsrc name=cam2 location=rtmp://127.0.0.1/live/cam2 ! 
flvdemux name=demux1
demux1.video ! identity name=v_delay1 signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! queue ! comp.sink_1

nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=10 sink_1::ypos=10 ! 
queue ! identity name=v_delay0 signal-handoffs=TRUE ! nvvidconv interpolation-method=5 ! 
queue ! identity name=v_delay1 signal-handoffs=TRUE ! nvvidconv interpolation-method=5 ! 
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! 
mpegtsmux name=mux ! appsink name=appsink

demux0.audio ! aacparse ! avdec_aac ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! 
audioconvert ! opusenc bitrate=320000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux.

Current GStreamer Configuration: Current GStreamer Configuration:

Update 1: I tried @SeB's Solution but it did not work:

Here are some screenshots showing the process:

videotestsrc on port 4953:

Port 4953

videotestsrc on port 4954:

Port 4954

full test pipeline:

full test pipline

Update 2: The Solution:

By Utilizing @SeB's answer and tinkering with it a bit, I was able to take two rtmpsrc's and compose them together, then send it to that same rtmp server under a different key, and use the rtmp pipeline that ships with the belacoder.

During my testing this only works if you follow the belabox tutorial, and not with the pre-made image.

Here is the pipeline that I used:

gst-launch-1.0 -v \
 rtmpsrc location=rtmp://127.0.0.1/live/cam1 ! flvdemux name=demux0 \
   demux0. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_0 \
   demux0. ! queue ! audio/mpeg ! mux. \
 rtmpsrc location=rtmp://127.0.0.1/live/cam2 ! flvdemux name=demux1 \
   demux1. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=YUY2,width=800,height=448,pixel-aspect-ratio=1/1 ! videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_1 \
 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=808,sink_1::height=456 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' \
   ! nvv4l2h264enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! h264parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. \
 flvmux name=mux ! rtmpsink location='location=rtmp://127.0.0.1/live/cam3 live=1'

Then I just edited the rtmp pipeline that comes with belacoder to pull from /cam3.

Here it is working in OBS Studio using belaUI + belacoder via SRTLA: obs-demo

This is the pipeline I used in belaUI/belacoder:

rtmpsrc location=rtmp://127.0.0.1/live/cam3 ! 
flvdemux name=demux 

demux.video ! identity name=v_delay signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! 
textoverlay text='' valignment=top halignment=right font-desc="Sans, 10" name=overlay ! queue ! 
nvvidconv interpolation-method=5 ! 
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. 

demux.audio ! aacparse ! avdec_aac ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! 
audioconvert ! voaacenc bitrate=128000 ! aacparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux. 

mpegtsmux name=mux ! 
appsink name=appsink

My settings are unique to the rtmp server I have running on my belabox (Jetson-nano) so keep that in mind.

Here is the final pipeline selected in the belaUI: belaui-pipeline-demo

Once you have it selected all you have to do is hit start and you can utilize all of the internet connections that are connected to the belabox: enter image description here

Please keep in mind this is really finicky, if one of your rtmps sources crap out it ruins the whole pipeline, so this works best when all rtmp sources are in a local environment, and you have the gts-launch pipeline running as a service.

If you want more information about the open-source DIY project belabox or would like to contact me, check out my profile links @ https://stackoverflow.com/users/3331416/b3ck

Solution

  • Just tried simulating your sources with (I don't have a RTMP server, but should be straight forward to try adapting):

    # Cam 1 1920x1080@30fps with audio
gst-launch-1.0 -e videotestsrc ! video/x-raw,format=NV12,width=320,height=240,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080,pixel-aspect-ratio=1/1' ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=mux    audiotestsrc ! audioconvert ! voaacenc ! queue ! mux.   mux. ! tcpserversink port=4953

    # Cam2 with 800x448@30fps
gst-launch-1.0 -e videotestsrc pattern=ball ! video/x-raw,format=NV12,width=320,height=240,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=800,height=448,pixel-aspect-ratio=1/1' ! nvv4l2h264enc ! h264parse ! queue ! flvmux ! tcpserversink port=4954

    Then, this should output video and audio:

    gst-launch-1.0 -v \
 tcpclientsrc port=4953 ! flvdemux name=demux0 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_0 \
 tcpclientsrc port=4954 ! flvdemux name=demux1 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=YUY2,width=800,height=448,pixel-aspect-ratio=1/1 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_1 \
 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=800,sink_1::height=448 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! autovideosink \
 demux0. ! queue ! audio/mpeg ! decodebin ! audioconvert ! audioresample ! autoaudiosink

    If ok, you can H265 encode composed video (note that here adding videobox the second frame will now have size 808x456) and forward mpeg audio with:

    gst-launch-1.0 -v \
 tcpclientsrc port=4953 ! flvdemux name=demux0 \
   demux0. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_0 \
   demux0. ! queue ! audio/mpeg ! tsmux. \
 tcpclientsrc port=4954 ! flvdemux name=demux1 \
   demux1. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=YUY2,width=800,height=448,pixel-aspect-ratio=1/1 ! videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_1 \
 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=808,sink_1::height=456 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' \
   ! nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! tsmux. \
 mpegtsmux name=tsmux ! appsink name=appsink