Camera2API adding ImageReader as a target surface laggs my camera preview

Question

I'm trying to process frames with the use of ImageReader while also displaying the video camera preview on a SurfaceView. If the only target surface I add is the SurfaceView the preview is smooth, but if I add an ImageReader as a second target the preview starts to lag heavily. Why could that be? I tried to create a HandlerThread and a Handler and use those, but it didn't change a thing. The only think I'm doing currently is to acquire and then close the next image.

public void startBackgroundThread() {
    handlerThread = new HandlerThread("Image Processing Thread");
    handlerThread.start();
    handler = new Handler(handlerThread.getLooper());
}

Configuring the camera here:

public void configureCamera() {
    try {
        CameraCharacteristics characteristics = cameraManager.getCameraCharacteristics(cameraId);
        StreamConfigurationMap configs = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);

        Size[] sizes = configs.getOutputSizes(ImageFormat.YUV_420_888);
        imageReader = ImageReader.newInstance(sizes[0].getWidth(), sizes[0].getHeight(), ImageFormat.YUV_420_888, 1);

        Range<Integer>[] ranges = characteristics.get(CameraCharacteristics.CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES);
        fpsRange = ranges[ranges.length - 1];

        imageReader.setOnImageAvailableListener(this, handler);

        cameraSurfaceView.getHolder().setFixedSize(sizes[0].getWidth(), sizes[0].getHeight());
    } catch (CameraAccessException | NullPointerException e) {
        e.printStackTrace();
    }
}

Starting the camera preview:

private void startCamera() {
    try {
        cameraManager.openCamera("0", new CameraDevice.StateCallback() {
            @Override
            public void onOpened(@NonNull CameraDevice camera) {
                cameraDevice = camera;

                try {
                    cameraDevice.createCaptureSession(Arrays.asList(cameraSurfaceView.getHolder().getSurface(), imageReader.getSurface()),
                            new CameraCaptureSession.StateCallback() {
                                @Override
                                public void onConfigured(@NonNull CameraCaptureSession session) {
                                    captureSession = session;
                                    try {
                                        requestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);

                                        requestBuilder.addTarget(cameraSurfaceView.getHolder().getSurface());
                                        requestBuilder.addTarget(imageReader.getSurface());

                                        requestBuilder.set(CaptureRequest.FLASH_MODE, CaptureRequest.FLASH_MODE_TORCH);
                                        requestBuilder.set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, fpsRange);

                                        captureRequest = requestBuilder.build();
                                        cameraReady = true;

                                        captureSession.setRepeatingRequest(captureRequest, null, handler);

                                        onStartButtonClick(startButton);
                                    } catch (CameraAccessException e) {
                                        e.printStackTrace();
                                    }
                                }

                                @Override
                                public void onConfigureFailed(@NonNull CameraCaptureSession session) {

                                }
                            }, null);
                } catch (CameraAccessException e) {
                    e.printStackTrace();
                }
            }

And here comes the ImageReader's OnImageAvailableListener (which is my main activity currently):

@Override
public void onImageAvailable(ImageReader reader) {
    reader.acquireNextImage().close();
}

These code snippets all reside in my main activity currently. The phone that I am using is a Motorola Moto X Play for testing purposes. The cameraSurfaceView is just a simple SurfaceView with no customizations whatsoever.

Answer 1

It's a little hard to see form the indentations, but it looks like you are running the camera handler on the current thread from the code snippet.

The last parameter in cameraManager.openCamera tells it what thread to use - if it is null it just uses the current thread. From the Android documentation:

Parameters cameraId String: The unique identifier of the camera device to open

callback CameraDevice.StateCallback: The callback which is invoked once the camera is opened

handler Handler: The handler on which the callback should be invoked, or null to use the current thread's looper.

If you look in the example on GitHub for Camera2Basic () you can see they specify a separate handler:

manager.openCamera(mCameraId, mStateCallback, mBackgroundHandler);

This is started on a separate thread:

    /**
     * Starts a background thread and its {@link Handler}.
     */
    private void startBackgroundThread() {
        mBackgroundThread = new HandlerThread("CameraBackground");
        mBackgroundThread.start();
        mBackgroundHandler = new Handler(mBackgroundThread.getLooper());
    }