I'm trying to create an app that uses Camera2 to both detect QR codes and record a video.
I have many outputs, and since I cannot attach everything at once to the Camera, I needed to do both my frame processing (QR Code or whatever) and my media recording in one single Camera Output.
I decided to go for an ImageReader, where I receive my Images which I can then process, and then pass them through to another output surface, in this case the MediaRecorder's Surface, using an ImageWriter. So:
ImageReader -> do some ML processing on the Image -> ImageWriter -> MediaRecorder
My current setup looks like this:
class VideoPipeline(width: Int,
height: Int,
format: Int): ImageReader.OnImageAvailableListener {
private var passThroughImageWriter: ImageWriter? = null
init {
imageReader = ImageReader.newInstance(width, height, format, 5)
imageReader.setOnImageAvailableListener(this, videoThreadHandler)
setupCamera(imageReader.surface, previewSurface, photoOutputSurface)
}
fun startRecording() {
val recorderSurface = MediaCodec.createPersistentInputSurface()
val recorder = MediaRecorder(...)
recorder.setVideoSource(MediaRecorder.VideoSource.SURFACE)
// a bunch of other setup code for media recorder
recorder.setInputSurface(surface)
recorder.prepare()
recorder.start()
passThroughImageWriter = ImageWriter.newInstance(recorderSurface, 5)
}
override fun onImageAvailable(reader: ImageReader) {
val image = reader.acquireLatestImage()
if (passThroughImageWriter != null) {
// Moves the Image to the ImageWriter, no need to close
passThroughImageWriter?.queueInputImage(image)
} else {
// Nothing else happening here, close image.
image.close()
}
}
}
But this just fails every time, no matter what format I use (Tried ImageFormat.YUV_420_888, ImageFormat.PRIVATE and PixelFormat.RGB888):
VideoPipeline D --> writing 1280 x 960 34 frame..
GraphicBufferSource D got buffer with new dataSpace #104
GraphicBufferSource W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
VideoPipeline D --> writing 1280 x 960 34 frame..
GraphicBufferSource W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
RecordingSession E MediaRecorder Error: 268435556 (-1007)
RecordingSession I Stopping RecordingSession..
VideoPipeline D --> writing 1280 x 960 34 frame..
GraphicBufferSource W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
MediaRecorder E stop failed: -1007
AndroidRuntime D Shutting down VM
AndroidRuntime E FATAL EXCEPTION: main
Process: com.mrousavy.camera.example, PID: 6406
java.lang.RuntimeException: stop failed.
at android.media.MediaRecorder.stop(Native Method)
at com.mrousavy.camera.core.RecordingSession.stop(RecordingSession.kt:103)
at com.mrousavy.camera.core.RecordingSession._init_$lambda$0(RecordingSession.kt:75)
at com.mrousavy.camera.core.RecordingSession.$r8$lambda$enBPf4dA0r-MTevnd6ChW75ZCDg(Unknown Source:0)
at com.mrousavy.camera.core.RecordingSession$$ExternalSyntheticLambda0.onError(Unknown Source:2)
at android.media.MediaRecorder$EventHandler.handleMessage(MediaRecorder.java:1615)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:201)
at android.os.Looper.loop(Looper.java:288)
at android.app.ActivityThread.main(ActivityThread.java:7872)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936)
Any ideas why?
Full code:
JFYI: Before I used ImageWriter, I had an OpenGL pipeline that just streamed into the MediaRecorder's surface using OpenGL Surfaces, that worked perfectly fine. So I think the MediaRecorder is set up properly, it's gotta be something with the ImageWriter imo.
Here's the diff where I switched from the OpenGL pipeline to ImageWriter: 4e96eb7 (reason being that OpenGL is always RGB, and I want PRIVATE/YUV frames)
I'd appreciate any pointers/help here!
When directly connecting the camera to the video encoder, the buffers are allocated with flags indicating they will be consumed by the encoder. This flag is used by the camera HAL and graphics buffer allocators to select appropriate formats/layouts/etc for the buffers.
When you use an ImageReader like this, only a flag for 'CPU use' is set; when you then try to send the buffer to the encoder, it likely errors out when trying to access the buffer from the hardware encoder.
You can try using this variant of the ImageReader constructor: https://developer.android.com/reference/android/media/ImageReader?hl=en#newInstance(int,%20int,%20int,%20int,%20long)
and pass in the USAGE_VIDEO_ENCODE and USAGE_CPU_READ_OFTEN flags. Note that this isn't a guaranteed-to-work combination, unfortunately. That said, I'd expect it to likely work with YUV_420_888 as the format. (PRIVATE format would mean you couldn't read from it in app code, since you won't know the layout).
The most compatible option is to either have two output streams, one to the encoder and one to the ImageReader for ML. Or do the copying in the GPU like you did before.