I have a GStreamer pipeline running on a Raspberry Pi on my home's LAN that is multicasting a UDP video (h264) and audio (opus) stream.
gst-launch-1.0 -v rpicamsrc vflip=true hflip=true \
name=src preview=0 fullscreen=0 bitrate=10000000 \
annotation-mode=time annotation-text-size=20 \
! video/x-h264,width=960,height=540,framerate=24/1 \
! h264parse \
! rtph264pay config-interval=1 pt=96 \
! queue max-size-bytes=0 max-size-buffers=0 \
! udpsink host=224.1.1.1 port=5001 auto-multicast=true\
alsasrc device=plug:dsnooped provide-clock=false \
! audio/x-raw,rate=16000 \
! audiorate \
! audioconvert \
! audioresample \
! opusenc \
! rtpopuspay \
! queue max-size-bytes=0 max-size-buffers=0 \
! udpsink host=224.1.1.1 port=5002 auto-multicast=true
I'm also using GStreamer to receive the audio and video streams
VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,payload=(int)96"
AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)48000,encoding-name=(string)OPUS"
gst-launch-1.0 -v udpsrc address=224.1.1.1 port=5001 caps=$VIDEO_CAPS \
! queue \
! rtph264depay \
! h264parse \
! mpegtsmux name=mux \
! hlssink location="/var/www/picam-viewer/hls/%06d.ts" playlist-location="/var/www/picam-viewer/hls/list.m3u8" max-files=5 playlist-length=1 target-duration=5
udpsrc address=224.1.1.1 port=5002 caps=$AUDIO_CAPS \
! queue \
! rtpopusdepay \
! opusdec caps="audio/x-raw,rate=48000,channels=2" ! audioconvert ! voaacenc ! aacparse \
! mux.
On the receiving side, I have tried many variations for the 2nd to last line (decoding Opus, converting to AAC), but in all cases I end up with HLS where the video works as expected, but there is no audio.
This is the verbose output I get from GStreamer when running the receiving pipeline:
(.venv) pi@picroft:~ $ sudo ./BabySpiCroft-Setup-Files/GStreamer/receive-stream-to-hls.sh
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, parsed=(boolean)true
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:sink_65: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, parsed=(boolean)true
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, width=(int)960, height=(int)540, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)4
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:sink_65: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal, width=(int)960, height=(int)540, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)4
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:src: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/GstHlsSink:hlssink0/GstMultiFileSink:multifilesink0.GstPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188
/GstPipeline:pipeline0/MpegTsMux:mux.GstPad:src: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
/GstPipeline:pipeline0/GstHlsSink:hlssink0/GstMultiFileSink:multifilesink0.GstPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
/GstPipeline:pipeline0/GstHlsSink:hlssink0.GstGhostPad:sink: caps = video/mpegts, systemstream=(boolean)true, packetsize=(int)188, streamheader=(buffer)< 47400030a600ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000b00d0001c100000001e020a2c32941, 474020308b00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0002b0280001c10000e041f00c050448444d5688040ffffcfc1be041f00a050848444d56ff1b443f5a3175c0 >
I am unable to tell if there's anything useful in this output. I suspect there's some parameter that needs to be set properly that I am missing, I just don't know what.
Thanks!
This receiving pipeline did the trick (shoutout to u/thaytan on the GStreamer subreddit!)
sudo rm /var/www/picam-viewer/hls/*.ts
sudo rm /var/www/picam-viewer/hls/*.m3u8
VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,payload=(int)96"
AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)48000,encoding-name=(string)OPUS"
sudo gst-launch-1.0 -v udpsrc address=224.1.1.1 port=5002 caps=$AUDIO_CAPS \
! rtpopusdepay \
! opusdec \
! audioconvert \
! avenc_aac \
! queue \
! combine_to_hls.audio \
udpsrc address=224.1.1.1 port=5001 caps=$VIDEO_CAPS \
! rtph264depay \
! h264parse \
! queue \
! hlssink2 location="/var/www/picam-viewer/hls/%06d.ts" playlist-location="/var/www/picam-viewer/hls/list.m3u8" max-files=5 playlist-length=3 target-duration=2 name=combine_to_hls
The key seemed to be switching from hlssink to hlssink2 which took care of the muxing for me.