Search code examples
actionscript-3osmfm3u8closed-captions

Capture or decode embedded CEA-608 captions in HLS MP4 video in Flash Player

Request

In Flash/AS3 how do you decode or capture CEA-608 closed-captions embedded in an mp4? I can't seem to get even a hint of the caption data nor can I find any documentation that can point me in the right direction. Any documentation, examples, or ideas would be super helpful.

Specifics

I'm building a video-player in AS3 using OSMF. I can't seem to find /any/ documentation on accessing CEA-608 closed-captions embedded in mp4 chunks in an m3u8 video.

The OSMF CaptioningPlugin requires an external XML file so that won't do; I'm looking for integration details for embedded captioning tracks.

I've tried attaching onTextData, onMetaData, onCaptionData, onTextRR handlers and listeners to the OSMF Netstream with absolutely no luck (like... none of these events or handlers ever fire or return anything).

private function onTraitAdd ($e:MediaElementEvent) : void { var mediaElement: MediaElement = ($e.target as MediaElement);

    switch ($e.traitType) {
        case MediaTraitType.LOAD:
            _netStreamLoadTrait = mediaElement.getTrait(MediaTraitType.LOAD) as NetStreamLoadTrait;
            _netStreamLoadTrait.addEventListener(LoadEvent.LOAD_STATE_CHANGE, onNetStreamLoaded);
            break;
    }
}

private function onNetStreamLoaded ($e:LoadEvent) : void {
    var netStream:NetStream = _netStreamLoadTrait.netStream;
    netStream.client.addHandler("onTextData", onTextData);
    netStream.client.addHandler("onCuePoint", onTextData);
    netStream.client.addHandler("onMetaData", onTextData);
    netStream.client.addHandler("onCaptionData", onTextData);
    netStream.client.addHandler("onTextRR", onTextData);
    netStream.client.addHandler("onCaptionInfo", onTextData);


    netStream.addEventListener("onTextData", onTextData);
    netStream.addEventListener("onCuePoint", onTextData);
    netStream.addEventListener("onMetaData", onTextData);
    netStream.addEventListener("onCaptionData", onTextData);
    netStream.addEventListener("onTextRR", onTextData);
    netStream.addEventListener("onCaptionInfo", onTextData);

    netStream.addEventListener(NetStatusEvent.NET_STATUS, onNetStreamStatus);
    netStream.addEventListener(NetDataEvent.MEDIA_TYPE_DATA, onStreamData);

}

I can't tell if the issue is with my OSMF implementation (maybe I'm listening to the wrong NetStream), or if the issue is that there's no way to get this data out of the video.

Example files:

http://stream.flowplayer.org/big_buck_bunny_with_captions.mp4 http://now.video.nfl.com/i/captiontest/closedcaptiontest_,350k,550k,.mp4.csmil/master.m3u8 (this example file is more complex though because it requires an HLS plugin)

Other

  1. I tried using OSMFCCDecoder.swc (which was rather hard to find, uploaded here). There's very little documentation and no information on the expected result.

  2. Also decompiled JWPlayer to see how they handle captions, they parse-out the byteArray ref.

  3. Steps to see captions playing in JWPlayer

jwplayer("container_wrapper")
    .setup({
        file: "http://now.video.nfl.com/i/captiontest/closedcaptiontest_,350k,550k,.mp4.csmil/master.m3u8"
    });
Solution

  • HLS Streams can contain subtitles in a few different formats:

    • CEA608
    • WebVTT
    • ID3 Tags CEA608 works on all Apple iOS devices from 3.0 onwards but is difficult to support because it's a character based format. WebVTT is supported in newer iOS devices but is text based and easer to decode.

    CEA608 is embedded in H264 SEI NALU as specified in ANSI/SCTE 128 2010 Section 8. So in order to decode it you would need to partially decode the H.264 bitstream. The link you provided http://help.adobe.com/en_US/adobemediaserver/devguide/WS5262178513756206-55daa065139e25f4596-8000.html seems to describe a library for decoding CEA608 and I believe the expected result will be text rendered over the video.

    It's hard work to decode CEA608 yourself as this is a character based format and might include letters, numbers, Spanish, backspace, color and position markers. You can find the CEA608 standard here: http://www.ce.org/Standards/Standard-Listings/R4-3-Television-Data-Systems-Subcommittee/Line-21-Data-Service.aspx You might also need to look at the MPEG2 Transport Stream specifications for the method to decode the H264 NALUs.