I'm building a FPV robot, I want to control it with a with a webbrowser over a local wi-fi connection.
I'm using a raspberry pi 3B+ with Raspbian Stretch. I built my own motor control and power regulator hat.
After lots of research testing, I decided to use node.JS as http server and socket.io to provide a low latency bidirectional communication with my robot. This stack achieve about 7ms of latency.
I need to stream low latency video from an USB camera attached to the RPI to the browser. My target is to achieve at least 640x480 resolution at 10FPS with 50ms of latency or better. I'm happy sacrificing visual fedelity to get a quicker response from my robot.
If possible I would like to stream in UDP to improve the reliability of the stream.
If possible I would like to stream a video that modern webbrowsers can natively decode. I'd like to use a H264 codec and the HTML5 video tag. I can fall back to use a javascript player if there is no other option.
I did an extensive research and tried many tools.
Amongst other, I tried VLC, mjpg streamer, gstreamer and raspivid. A few times I got to a stream the webbrowser could view, but at best I got a latency of 700ms at 320x240. Very very far from my target.
Currently I'm looking into WebRTC solutions.
I'd like suggestions for NODE.JS packages or other solutions to provide a UDP H264 video stream that can be decoded by an HTML5 video tag with a target latency of 50ms.
Thanks
Thanks for your answers! I'll keep updating this question and I'll post the solution once it works.
PUSH INDIVIDUAL FRAMES
I tried a different approach by pushing individual 200KB 640x480 jpg frame through websocket and I got a latency of about 190ms. I can probably do a lot better by reusing objects but I'm putting this attempt in hold for now.
While researching WebRTC I found a stack that looked easy enough. Server side it uses V4L2 as driver, FFMPEG to transcode into an MPEG1 http stream with TS encapsulation locally, node js to flip the stream into a websocket. Client side there is a javascript that decode the MPEG1 TS stream and paint a canvas object into the HTML page.
It achieves 640x480@20FPS with 240mS of latency. Good enough for an MVP, but I'll keep working to get it down. Code in the answer.
I’d like suggestions for NODE.JS packages or other solutions to provide a UDP H264 video stream that can be decoded by an HTML5 video tag with a target latency of 50ms.
That’s Almost certainly not possible in that configuration.
If you drop the video tag requirement, and use just straight WebRTC in the browser, you may be able to get down to about 150ms.
