Search code examples
opencvudpwebrtcjanus-gateway

Reconstruct image from RTP packets

I am trying to stream a user's webcam over the network to a C-based server. I have used Janus gateway.

I created a small plugin which is heavily based on the echotest demo example : I have my browser connecting to my janus server via WebRTC technology and I have it stream the user's webcam.

On the server side, I have janus_incomming_rtp function which gives me a char * buffer and int length. Upon inspection, the buffer of data that's incomming is about the length of the MTU : each frame of my video is sent upon several RTP packets.

I have inspected a bit the header by following this wikipedia page but I don't know how to reconstruct the image from that stream of UDP RTP packets. Ideally, I'd like to pass the stream to openCV to do realtime image processing.

I have heard of gstreamer, but I don't undertstand what it is nor how it could help me ; besides I don't know if openCV has any built in functions to "reconstruct" the images ? And I don't know in which format the video frames are being encoded : the PT (Payload Type) seems to be 116 which is defined as "dynamic" but I have no idea what it means.

Any help ?

Solution

  • Here are some guiding steps for handling the SRTP packets to decode them.

    1. Make sure that rtp and RTCP are not multiplexed, you can remove that option from the SDP
    2. Decrypt the SRTP packet to raw RTP, you will need access to the key exchange(not sure if you are already doing this but all media is encrypted and keys exchanged using DTLS and must be decrypted before handling)
    3. Grab your media payload type and match it against the media from SDP(you can see from the RTPMAP in the SDP what media is what payload)
    4. Remove the RTP Payload from the packet(Gstreamer has RtpDepay plugins for most common payloads, including VP8) and decode the stream. Quick example of command line pipelines using vp8
    5. Now you have a raw video/audio packet that can be displayed.

    SDP:

    • If RTCP and RTP are being multiplexed you will see the line a=rtcp-mux and you will see that the port in a=rtcp:50111 IN IP4 <address> and the candidate media ports will be the same.
    • If the media itself is being multiplexed you will see a=group:BUNDLE audio video

    SRTP:

    • Janus handles the DTLS exchange already and it seems that it may already decrypt the rtp before sending it but it does not look like it accounts for multiplexed rtp/rtcp and media.
    • Here is a quick and dirty SRTP decrypter that works when you pass it the MasterKey that is exchanged in DTLS.

    GStreamer:

    • You may want to look into the GstAppSrc which allows you to char arrays into a gstreamer pipeline for decoding and you can push it to another udp port to grab it with OpenCV.
    • Here is some example code from a websocket server I wrote that will grab raw media and push it to a pipeline. This example is not exactly what you want to do(it does not grab the RTP but instead raw media frames from the webpage) but it will show you how to use AppSrc.