Search code examples
htmljpeggstreamer

JPEG live stream in html slow

From a raw video source I'm trying to stream jpeg images to html as fast as possible in a embedded platform/board using linux.

At the gstreamer side I can see that the jpeg image is updated at ~37fps the pipeline looks like this:

appscr -> videoconvert -> jpegenc -> multifilesink

based in this question I created the next embedded html:

<!DOCTYPE html>
<html>
    <head>
        <meta charset='UTF-8' />
        <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
        <script src="app.js"></script>
    </head>
    <body>  
        <img id="snapshot" src="snapshot.jpeg"/>
    </body>
</html>

and the java script:

$(function() {
    function refreshImage(){
    $("#snapshot").attr("src", 'snapshot.jpeg?' + Math.random());
    setTimeout(refreshImage, 20);
    }
    refreshImage();
})

Opening a web browser from a PC and typing the platform/board IP I can see the video stream, but the problem is that the image is updated too slow and I would expect a more fluid/fast video given the source frame rate (37fps).

does anyone know what could be the reason why the update is slow?

Solution

  • I think this deserves proper analysis since it is interesting subject (at least for me).

    Testing environment

    I completely replicated scenario on 2 PCs within same LAN.

    PC 1 is creating jpeg images from live stream with following pipeline

    gst-launch-1.0 -v rtspsrc location="rtsp://freja.hiof.no:1935/rtplive/_definst_/hessdalen03.stream" \
! rtph264depay ! avdec_h264 \
! timeoverlay halignment=right valignment=bottom \
! videorate ! video/x-raw,framerate=37000/1001 ! jpegenc ! multifilesink location="snapshot.jpeg"

    and serving index.html, app.js and (endlessly updated) snapshot.jpeg with python's simple http server

    python -m SimpleHTTPServer 8080

    PC 2 is accessing index.html using Chrome browser (with developer tools window) and showing images.

    For testing purposes

    • I've added timeoverlay in gstreamer pipeline that adds timestamp on each image in right bottom corner.
    • I increased refresh period in JS function on 1000 ms.

    Analysis of test results

    Here is browser's network log

    enter image description here

    Time column shows periods (in ms) that browser spends to fetch (download) one image from server. Those periods are not always the same with average of ~100ms for images with size ~87KB.

    Fetch time interval actually includes:

    • interval that HTTP GET needs to reach server from browser,
    • interval that server needs to read image from disk and send it back as HTTP response,
    • interval that HTTP response needs to reach browser.

    1st and 3rd interval are directly depend on "internet" environment: if browser is "farther" away from server interval will be greater.

    2nd interval is proportional to server "speed": how fast server can read images from disk and handle HTTP request/response

    There is another interval proportional to "speed" of PC which runs browser: how fast PC can handle HTTP GET request/response and re-render image.

    Conclusion

    There are many unavoidable delay intervals that depend on testing environment - internet and capabilities of server machine and client machine with browser - and your code in browser is executing as fast as it possibly can.

    In any case, 37 fps sounds like some live stream video. There are specialized protocols to stream video that can be shown in browser (e.g. MPEG DASH or HLS) by serving video chunk-by-chunk (where each chunk contains many video frames).