I'm writing some code in Android, I have to make a face detection with camera2 API. My code is based on android-camera2-api-face-recon samples and I don't know why but when I use face.getBound(), the values it return are out of the view.
This is for Android studio, in JAVA, this code is supposed to draw a rectangle around a head and it recognize there is a head, but the rectangle is drawn anywhere and don't follow the head.
private void process(CaptureResult result) {
Integer mode = result.get(CaptureResult.STATISTICS_FACE_DETECT_MODE);
Face[] faces = result.get(CaptureResult.STATISTICS_FACES);
mFaceDetectionMatrix=new Matrix();
if (faces != null && mode != null) {
if (faces.length ==1) {
for(int i = 0; i < faces.length; i++) {
if (faces[i].getScore() > 50) {
int left = faces[i].getBounds().left;
int top = faces[i].getBounds().top;
int right = faces[i].getBounds().right;
int bottom = faces[i].getBounds().bottom;
float points[] = {(float)bottom, (float)right, (float)top, (float)left};
Log.i("Test", "faces : " + faces.length + " , mode : " + mode + " left " + left + " top " + top + " right " + right);
Rect uRect = new Rect(left, top, right, bottom); //The function I'm supposed to use
//Rect uRect = new Rect(+2000-bottom, 1500-left, +1500-top, 3500-right) the function I'm using in order to fix this problem
The rectangle move to the left or the right when the head move down or upward and vice versa. As you can see, I'm trying some unusable thing to fix it and I have to completely change the entrees in Rect.
I would like to fix this problem properly and understand why it doesn't work, and if I can't I would like to change the constants I put in Rect in order to be able to use this code on all devices.
Do you have some ideas ?
In order to control sensor orientation, I use this code :
int orientationOffset;
orientationOffset = characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION);
Rect activeArraySizeRect = characteristics.get(CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE);
//Face Detection Matrix
//mFaceDetectionMatrix = new Matrix();
// TODO - I guess that is not enough if we have a landscape layout too...
mFaceDetectionMatrix.setRotate(orientationOffset);
Log.i("Test", "activeArraySizeRect1: (" + activeArraySizeRect + ") -> " + activeArraySizeRect.width() + ", " + activeArraySizeRect.height());
Log.i("Test", "activeArraySizeRect2: " + mPreviewSize.getWidth() + ", " + mPreviewSize.getHeight());
float s1 = mPreviewSize.getWidth() / (float)activeArraySizeRect.width();
float s2 = mPreviewSize.getHeight() / (float)activeArraySizeRect.height();
//float s1 = mOverlayView.getWidth();
//float s2 = mOverlayView.getHeight();
boolean mirror = true; // we always use front face camera
boolean weAreinPortrait = true;
mFaceDetectionMatrix.postScale(mirror ? -s1 : s1, s2);
//mFaceDetectionMatrix.postRotate(orientation);
if (mSwappedDimensions) {
mFaceDetectionMatrix.postTranslate(mPreviewSize.getHeight(), mPreviewSize.getWidth());
} else {
// TODO - ...
}
It's like... the rectangle is in landscape and the head in portrait.
Thank you
[EDIT] I found a way to fix it ! In the OverlayView class,in onDraw, I add this :
canvas.scale(-1, 1, getWidth()/2, getHeight()/2);
and I modified OverlayView in fragment_camera2_basic.xml while adding a android:rotation="90" and now the rectangle move like it have to.
It has transformed the problem, now, depending on used device, the rectangle don't have the same position.
I have modified top, left, right and bottom like that :
int left = (int)((float)(faces[i].getBounds().left)/MGpix.getWidth()*mTextureView.getHeight());
int top = (int)((float)(faces[i].getBounds().top)/MGpix.getHeight()*mTextureView.getWidth());
int right = (int)((float)(faces[i].getBounds().right)/MGpix.getWidth()*mTextureView.getHeight());
int bottom =(int)((float)(faces[i].getBounds().bottom)/MGpix.getHeight()*mTextureView.getWidth());
MGpix is equal to characteristics.get(CameraCharacteristics.SENSOR_INFO_PIXEL_ARRAY_SIZE);
Now it's near to work, but the rectangle is, depending on the device, either around the head or next to, but it follow the head as it have. For example, on a Honor 5c, the rectangle is well placed, but with a huawey p9 lite, the up-left corner of the rectangle is on the nose. Any idea in order to fix it ?
How are you converting from the face rectangle coordinates, which are in the sensor active array coordinate system, to the coordinates of your preview View?
You need to account for the sensor's orientation as well as the current orientation of your application relative to the device's native orientation.
Edit: Your current calculation of s1/s2 isn't correct for a landscape sensor on a portrait device (which describes most phones), even for the case of your app being locked to portrait orientation.
You need to swap width/height of either preview or activeArray before the s1/s2 calculation if the orientation of your app and the sensor don't match.
A rotate after that will only swap two incorrectly calculated coordinates with each other.
The most robust option is to get the current display's orientation (which will be relative to the device's native orientation), and use that and the sensor orientation to determine the relative orientation of the sensor and your app, to decide when to swap coordinates. Something like:
int displayRotation = WindowManager.getDefaultDisplay().getRotation();
boolean displayIsRotated = (displayRotation == Surface#ROTATION_90) ||
(displayRotation == Surface#ROTATION_270)
int cameraOrientation = characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)
boolean cameraIsRotated = (cameraRotation == 90) || (cameraRotation == 270)
boolean flipAxes = displayIsRotated ^ cameraIsRotated;
if (flipAxes) {
float s1 = mPreviewSize.getWidth() / (float)activeArraySizeRect.height();
float s2 = mPreviewSize.getHeight() / (float)activeArraySizeRect.width();
} else {
float s1 = mPreviewSize.getWidth() / (float)activeArraySizeRect.width();
float s2 = mPreviewSize.getHeight() / (float)activeArraySizeRect.height();
}