Search code examples
androidcameraaspect-ratioandroid-camerax

Android CameraX seems to crop analysis images slightly for specific PreviewView sizes

Dependencies used:

implementation 'androidx.camera:camera-camera2:1.1.0-alpha04'
implementation 'androidx.camera:camera-lifecycle:1.1.0-alpha04'
implementation 'androidx.camera:camera-view:1.0.0-alpha24'

Pretend that for simplicity you're using a device with:

  • a portrait default orientation
  • 1080x1920 screen resolution
  • with ImageAnalysis configured to use 1080x1920 as frames analysis resolution

When a PreviewView is laid out with sizes close to 1080x1920 and FILL_CENTER as a scale type, we're getting analysis frames with the same contents as we see in the PreviewView - it's ok. When our PreviewView becomes a little bit more narrow, for example 700x1920 - it gets cropped parts of analysis frames: some parts of the frames on the left and right now look cropped - it's ok too.

But when we change the sizes of the PreviewView to make it look as landscape, for example - to 1080x500 there is an issue appears. Our PreviewView makes the frames from the camera look cropped from top and bottom - it's ok. But what's not ok - is that our analysis frames become actually cropped veeery slightly at left and right. That means I expect to see the same capture bounds horizontally both in PreviewView and frames (because due to scaling only vertical parts are cropped), but PreviewView actually shows a little bit more along its width rather than the corresponding frame, so it looks like the frame looses about 20px horizontally, what shouldn't occur, because due to frame (1080x1920) and PreviewView (1080x500) sizes, only vertical parts should be clipped.

Attaching an illustration: enter image description here

Do anyone encounter the same behaviour in the CameraX?

UPD: How did I find this out? - By saving frames (images) to files. Frame buffers are converted to cv::Mats with:

void nv21_buffer_to_bgr_mat(JNIEnv* env,
                            jbyteArray nv21_buffer,
                            jint width,
                            jint height,
                            cv::Mat& mat)
{

    // To YUV.
    jbyte* nv21_buffer_bytes = env->GetByteArrayElements(nv21_buffer, nullptr);
    cv::Mat yuv(height + height / 2, width, CV_8UC1, nv21_buffer_bytes);

    // To BGR.
    cvtColor(yuv, mat, cv::COLOR_YUV2BGR_NV21, 3);
    env->ReleaseByteArrayElements(nv21_buffer, nv21_buffer_bytes, 0);

}

UPD2: Maybe layout inspector images for the described above cases will be helpful: enter image description here enter image description here

Solution

  • The underlying camera2 API has a fairly strictly defined set of rules on how having multiple output resolutions works, in terms of cropping. The docs for the SCALER_CROP_REGION (digital zoom) control have useful diagrams about this; for your case, just assume the CROP_REGION covers the whole active array.

    On top of that, CameraX PreviewView applies further cropping to its input, since the camera device itself only supports a few sizes. So PreviewView will pick a supported resolution that has a reasonable aspect ratio, and then crop it if necessary when displaying it to fill your View layout area.

    So, the relationship between the ImageAnalysis and Preview use cases and their fields of view depends on both the resolutions CameraX selects for the use cases under the hood, and PreviewView's additional cropping.

    What's probably happening here is that your landscape PreviewView is selecting an underlying resolution that's wider than ImageAnalysis is selecting. You could check what's actually selected via adb shell dumpsys media.camera if you have developer access to a device, while your app is running. That command dumps out a lot of info, including exactly what CameraX has configured the camera device to do.

    You could try using the setTargetAspectRatio or the setTargetResolution methods on Preview.Builder to match the aspect ratio of the ImageAnalysis; that should ensure that the PreviewView is only ever a crop of what ImageAnalysis receives.