As far as I know, MediaCodec return output buffer of image in YUV420 and then you can process with it as you like to... But in my case I need to convert YUV420 in BGR, it is quite expensive conversion.
So, question is if it is possible immediately (without conversion) get BGR from MediaCodec?
My code now
uint8_t *buf = AMediaCodec_getOutputBuffer(m_data.codec, static_cast<size_t>(status), /*bufsize*/nullptr);
cv::Mat YUVframe(cv::Size(m_frameSize.width, static_cast<int>(m_frameSize.height * 1.5)), CV_8UC1, buf);
cv::Mat colImg(m_frameSize, CV_8UC3);
cv::cvtColor(YUVframe, colImg, CV_YUV420sp2BGR, 3);
auto dataSize = colImg.rows * colImg.cols * colImg.channels();
imageData.assign(colImg.data, colImg.data + dataSize);
So, as you can see here
cv::Mat YUVframe(cv::Size(m_frameSize.width, static_cast<int>(m_frameSize.height * 1.5)), CV_8UC1, buf);
I am getting YUVframe from codec buffer and then here
cv::cvtColor(YUVframe, colImg, CV_YUV420sp2BGR, 3);
I make conversion
Conversion take a lot of time. For me performance is critical.
The frames come out of MediaCodec in whatever format the codec likes to work in. Most digital cameras and video codecs work with YUV format frames. The camera is required to use one of two specific formats when capturing still images, but MediaCodec is more of a free-for-all. If you want the data to be in a specific format, something has to do the conversion.
On Android you can get the hardware to do it for you by involving the GPU, which is required to accept whatever format MediaCodec decides to generate on that device. You latch each frame as an "external" texture by feeding it into a SurfaceTexture, and then tell GLES to render it in RGB on a pbuffer, which you can then access in different ways. For an example that uses glReadPixels(), see ExtractMpegFramesTest.
There are alternatives to glReadPixels() that rely on private native methods to access the pbuffer (e.g. this and this), but using non-public APIs is unwise unless you have complete control of the device.
There may be some newer approaches for accessing the pbuffer that I haven't used, e.g. using native hardware buffers seems potentially useful (or maybe not?).