EDIT: Nevermind, I solved my problem.
//------------------------------------------------------------------------------
I'm having some problems trying to do MJPEG stream decoding using Android MediaCodec API. What I'm trying to do is to stream jpeg frames from my webcam and display them using hardware accelerated decoding on my android device (currently I'm using Nvidia Shield Tablet, Android 6.0).
Right now I'm at the point where I receive a jpeg frame and I can display it on the screen by using BitmapFactory.decodeStream() function. However, what I want to do is to decode the image on a dedicated hardware so I can get a better performance and better battery life. In order to do so I want to use MediaCodec API with a 'video/mjpeg' deocoder which I'm pretty sure is available on my device (I checked the MediaCodecList and it's right under the name 'OMX.Nvidia.mjpeg.decoder').
I'm using my MediaCodec with a SurfaceView, pretty much in the same way it's done in 'Grafika' repo. (https://github.com/google/grafika)
This is the decoder initialization code, called in surfaceCreated() callback, completes without any exceptions:
private Surface testSurface;
private MediaCodec decoder;
//....
private void initMediaApi() throws IOException {
decoder = MediaCodec.createDecoderByType("video/mjpeg");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/mjpeg", 720, 480);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 25);
decoder.configure(mediaFormat, testSurface, null, 0);
decoder.start();
}
After that I have a function that is called every time a new jpeg frame is received. It takes two parameters: a byte array with jpeg image and the size in bytes of this image:
private void testMediaApi(final byte[] frameBuffer, final int length) throws IOException {
final int TIMEOUT_USEC = 10000;
long firstInputTimeNsec = System.nanoTime();
int inputBufIndex = decoder.dequeueInputBuffer(TIMEOUT_USEC);
if (inputBufIndex < 0) {
return;
}
ByteBuffer inputBuffer = decoder.getInputBuffer(inputBufIndex);
inputBuffer.clear();
inputBuffer.put(frameBuffer, 0, length);
decoder.queueInputBuffer(inputBufIndex, 0, length, 0, 0);
boolean outputDone = false;
while (!outputDone) {
outputDone = true;
MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo();
int decoderStatus = decoder.dequeueOutputBuffer(mBufferInfo, 1000000);
Log.d(LOG_TAG, "decoderStatus: " + decoderStatus);
if (decoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
// no output available yet
Log.d(TAG, "no output from decoder available");
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
// not important for us, since we're using Surface
Log.d(TAG, "decoder output buffers changed");
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = decoder.getOutputFormat();
Log.d(TAG, "decoder output format changed: " + newFormat);
} else if (decoderStatus < 0) {
throw new RuntimeException("unexpected result from decoder.dequeueOutputBuffer: " + decoderStatus);
} else { // decoderStatus >= 0
if (firstInputTimeNsec != 0) {
// Log the delay from the first buffer of input to the first buffer
// of output.
long nowNsec = System.nanoTime();
Log.d(TAG, "startup lag " + ((nowNsec - firstInputTimeNsec) / 1000000.0) + " ms");
firstInputTimeNsec = 0;
}
Log.d(TAG, "surface decoder given buffer " + decoderStatus + " (size=" + mBufferInfo.size + ")");
if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
Log.d(TAG, "output EOS");
outputDone = true;
}
boolean doRender = (mBufferInfo.size != 0);
// As soon as we call releaseOutputBuffer, the buffer will be forwarded
// to SurfaceTexture to convert to a texture. We can't control when it
// appears on-screen, but we can manage the pace at which we release
// the buffers.
decoder.releaseOutputBuffer(decoderStatus, doRender);
Log.d(TAG, "Reached EOS, looping");
decoder.flush(); // reset decoder state
}
}
}
As you can see I'm waiting 1 second to complete one frame decoding, which should be plenty. Unfortunately, after the first call I receive decoder status code -1, which means output data is still not available. After the second frame is queued I get status -3 'decoder output buffers changed' and then the whole thing crashes giving me this exception:
07-20 21:21:25.512 10096-10109/piotrek.androidfpvtest W/MediaCodec: ResourceManagerService died.
07-20 21:21:25.513 10096-10127/piotrek.androidfpvtest E/ACodec: OMX/mediaserver died, signalling error!
07-20 21:21:25.513 10096-10127/piotrek.androidfpvtest E/ACodec: signalError(omxError 0x8000100d, internalError -32)
07-20 21:21:25.513 10096-10126/piotrek.androidfpvtest E/MediaCodec: Codec reported err 0xffffffe0, actionCode 0, while in state 6
07-20 21:21:25.513 10096-10125/piotrek.androidfpvtest W/System.err: java.lang.IllegalStateException
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2622)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.testMediaApi(MainActivity.java:226)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.doReceive(MainActivity.java:161)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity.access$000(MainActivity.java:29)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at piotrek.androidfpvtest.MainActivity$1.run(MainActivity.java:54)
07-20 21:21:25.514 10096-10125/piotrek.androidfpvtest W/System.err: at java.lang.Thread.run(Thread.java:818)
If instead of continuously queuing frames I wait for the first one to be decoded the program gets into infinite loop with output looking like this:
07-20 21:58:29.970 10577-10641/piotrek.androidfpvtest I/OMXClient: Using client-side OMX mux.
07-20 21:58:30.110 10577-10640/piotrek.androidfpvtest I/MediaCodec: [OMX.Nvidia.mjpeg.decoder] setting surface generation to 10830849
07-20 21:58:30.115 10577-10641/piotrek.androidfpvtest I/ACodec: Enable timestamp filtering for Video Decoder
07-20 21:58:30.126 10577-10641/piotrek.androidfpvtest D/SurfaceUtils: set up nativeWindow 0xae47c708 for 720x480, color 0x106, rotation 0, usage 0x2b00
07-20 21:58:56.719 10577-10641/piotrek.androidfpvtest D/SurfaceUtils: set up nativeWindow 0xae47c708 for 720x480, color 0x147, rotation 0, usage 0x2b00
07-20 21:58:56.720 10577-10641/piotrek.androidfpvtest W/ACodec: [OMX.Nvidia.mjpeg.decoder] setting nBufferCountActual to 11 failed: -1010
07-20 21:58:56.720 10577-10641/piotrek.androidfpvtest W/ACodec: [OMX.Nvidia.mjpeg.decoder] setting nBufferCountActual to 10 failed: -1010
07-20 21:58:57.713 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:57.713 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:58:57.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -3
07-20 21:58:57.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoder output buffers changed
07-20 21:58:58.714 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:58.715 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:58:59.717 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:58:59.718 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:00.720 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:00.720 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:01.721 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:01.721 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:02.723 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:02.723 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:03.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:03.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:04.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:04.724 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:05.726 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:05.726 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
07-20 21:59:06.727 10577-10639/piotrek.androidfpvtest D/MediaAPITest: decoderStatus: -1
07-20 21:59:06.727 10577-10639/piotrek.androidfpvtest D/MediaAPITest: no output from decoder available
If anyone has any experience with this API I would be very thankful for any help!
I solved my problem myself. It turns out that this particular decoder doesn't support output to surface, so the solution is to get raw ByteBuffer and then use GLES to do YUV->RGB conversion and display it on the screen.