Search code examples
androidvideo-processingandroid-mediacodec

How can I tell what MediaFormats are supported for video encoding on an Android device?

I am developing a library for video/photo processing (add filters like Instagram/Snapchat). So far the core features work very well.

However, I am finding the processing of a video (re-encoding an input video) to be hugely frustrating. There seem to be a number of edge cases and device specific issues that prevent the library from working 100% of the time.

I would like to know how to select/create a MediaFormat that will work on a device.

Currently, I'm setting up the MediaFormat that will be used to encode a video as follows:

// assume that "extractor" is a media extractor wrapper, which holds a 
// reference to the MediaFormat of the input video
fun getOutputVideoFormat(): MediaFormat {
    val mimeType = MediaFormat.MIMETYPE_VIDEO_H263
    var width = -1
    var height = -1
    var frameRate = 30
    var bitrate = 10_000_000
    val colorFormat = MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface

    if (extractor.videoFormat.containsKey(MediaFormat.KEY_WIDTH)) {
        width = extractor.videoFormat.getInteger(MediaFormat.KEY_WIDTH)
    }

    if (extractor.videoFormat.containsKey(MediaFormat.KEY_HEIGHT)) {
        height = extractor.videoFormat.getInteger(MediaFormat.KEY_HEIGHT)
    }

    if(extractor.videoFormat.containsKey(MediaFormat.KEY_FRAME_RATE)){
        frameRate = extractor.videoFormat.getInteger(MediaFormat.KEY_FRAME_RATE)
    }

    if(extractor.videoFormat.containsKey(MediaFormat.KEY_BIT_RATE)){
        bitrate = extractor.videoFormat.getInteger(MediaFormat.KEY_BIT_RATE)
    }

    val format = MediaFormat.createVideoFormat(mimeType, width, height)
    format.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat)
    format.setInteger(MediaFormat.KEY_BIT_RATE, bitrate)
    format.setInteger(MediaFormat.KEY_FRAME_RATE, frameRate)
    format.setInteger(MediaFormat.KEY_CAPTURE_RATE, frameRate)

    // prevent crash on some Samsung devices
    // http://stackoverflow.com/questions/21284874/illegal-state-exception-when-calling-mediacodec-configure?answertab=votes#tab-top
    format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, width * height)
    format.setInteger(MediaFormat.KEY_MAX_WIDTH, width)
    format.setInteger(MediaFormat.KEY_MAX_HEIGHT, height)
    format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 0)
    return format
}

So far this works on all major devices that I have tested with, but there are some devices like the Samsung A5 that have been reported to fail silently using this format, and simply create corrupted output videos using an input video that works correctly on all other devices.

How can I tell if a MediaFormat will actually succeed on a given device?

The only logs I have from the Samsung A5 device indicate that when the MediaCodec sends through the "INFO_OUTPUT_FORMAT_CHANGED" signal, the following media format is returned: 

csd-1=java.nio.ByteArrayBuffer[position=0,limit=8,capacity=8], 
mime=video/avc,  
frame-rate=30,  
remained_resource=2549760,  
height=480,  
width=480,  
max_capacity=3010560, what=1869968451,  
bitrate=10000000,  
csd-0=java.nio.ByteArrayBuffer[position=0,limit=17,capacity=17]

This format seems invalid to me, given the fact the input video has a resolution of 1280x720

Solution

  • It turns out that my problem had nothing to do with the video codecs available on the device. The problem didn't come from MediaCodec or MediaFormat, but from MediaMuxer.

    I was processing video and audio by reading them out through a MediaExtractor, pushing that through a MediaCodec configured for decoding, processing that data, and then pushing processed data through a MediaCodec configured for encoding. Then I was pushing the encoded data to a MediaMuxer (and eventually writing it to a file). This is very similar to the DecodeEditEncodeTest found on https://bigflake.com/mediacodec/. I was only doing processing on the video track, but I was using a similar decode/encode method to take the audio from the input file and put it into the output file.

    I initially thought the problem was device specific, but it turns out that the issue was actually with the input. The video that was causing the issue with processing was very short - less than 2 seconds long. Decoding and re-encoding the audio was not working correctly with such a short video, and the MediaMuxer was not registering any audio frames. This is what was causing the final output to be corrupted.

    I found the following CTS test: https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/MediaMuxerTest.java, and its cloneMediaUsingMuxer method, which shows how to copy audio directly from a MediaExtractor into a MediaMuxer.

    I modified my processing method to (continue to) use the decode/edit/encode method for video, and to use the passthrough method demonstrated by the CTS test for writing audio. This solved the issue, and I was able to process the short video correctly.