Search code examples
javascriptvideogetusermediaamazon-rekognitionweb-mediarecorder

Convert video codecs to vp9 to mp4?

I am recording the video from the getUserMedia function in js with option

let options = {mimeType: 'video/webm;codecs=vp9'};
// here mediaStreamObj is stream returning from the getUserMedia
let mediaRecorder = new MediaRecorder(mediaStreamObj, options);

I want the codecs to be h264 I have also tried video/webm;codecs=h264 in this it is giving mime type as video/x-matroska image

video/mp4;codecs=h264 - not supported throws error

I need mimeType as mp4 and codecs h264.

As I am working on a video analysis (amazon recognition). and I want to use startLabelDetection in php for detecting the labels in the video and amazon only accept h264 format video.

https://docs.aws.amazon.com/aws-sdk-php/v3/api/api-rekognition-2016-06-27.html#startlabeldetection

Is there anyone I can convert the video mime-type to mp4 with js(preferrable) or php, so that I can use it with amazon rekognition for video analysis?

Expected mimetype:

enter image description here

You can check the mime type here https://www.metadata2go.com/

Solution

  • tl;dr You can't get MP4 from your browsers' MediaRecorder APIs. You can, I believe, get Amazon's Elastic Transcoder to convert webm to mp4. You should also try to feed your webm / h.264 files to Rekognition if you have not already done so.

    As you've realized, there are two parts to your requirement, boxing and video. We need a bit of vocabulary to answer your question in detail.

    Boxing

    First up is the so-called "boxing" or container format. webm is a format based on Matroska, based in turn on Extensible Binary Markup Language (EBML). EBML was developed for Matroska. It's like XML but takes less RAM and disk space because it's not nearly as verbose. webm itself is an open and royalty-free standard for media files.

    Another boxing format is MPEG-4's. It's almost identical to Apple's mov movie-file format and is called the mp4 format.

    WebRTC has its own boxing schemes adapted to datagram (instead of data stream) communications. Out of the scope of your question.

    Codec

    What goes in those webm or mp4 boxes? Audio and video data. (They can also contain other time-synchronized data streams like captions, as needed.)

    • h.264 also known as Advanced Video Coding, MPEG-4 Part 10, or AVC.
    • vp8, the open-source and patent free codec developed by On2 acquired by Google
    • h.265 also known as High Efficiency Video Coding or HEVC.
    • vp9, the more recent open source and patent free codec developed at Google.

    The latter two, newer, codecs require more compute cycles to compress. And h.265 is still subject to a significant patent burden. So browsers and other general-purpose software typically use the first two, h.265 and vp8.

    What can browsers handle?

    Chromium browsers (Google Chrome, Edge) and Firefox have the MediaRecorder API. They can generate webm-boxed data streams containing audio and/or video. They cannot generate mp4-boxed data streams.

    Those data streams from those browsers can (as of October 2020) contain VP8 or H.264-coded video. In the case of video data, the available MIME types are

    • video/webm; codecs="vp8" (for vp8)
    • x-matroska/webm; codecs="avc1.42E01E" or video/webm; codecs="avc1.42E01E" for H.264.

    Formally speaking you should use x-matroska/video rather than webm/video as the MIME type for video streams using H.264, even though either MIME type works in browsers. The webm standard requires its video codecs to be either vp8 or vp9, so video/webm; codecs="avc1.42E01E", formally speaking, does not conform to that standard.

    Javascript modules

    • npm ebml can unbox and box data into webm streams. I've used its unboxing capability in large scale production. It works.
    • npm mp4-box-encoding can box data into mp4 streams. I haven't used this.