Sceneform - Getting Vector3 position from screen pixel position?

Question

The Sceneform camera has the camera.setWorldPosition method which accepts a Vector3.

I have currently calculated the center point of where the user pinches the screen using the ScaleGestureDetector.

My question is, how can I calculate the Vector3 position I want to move to based a single point on the screen?

Let's say the user pinches near the middle of the screen --- the ratio for the X and Y coordinates calculated from the phone's display metrics are

X: 0.58122176, Y: 0.46196362

How do I convert it into a Vector3 so my SceneView camera can zoom in that position on the screen?

 private class ZoomGestureDetector extends ScaleGestureDetector.SimpleOnScaleGestureListener {
        @Override
        public boolean onScale(ScaleGestureDetector detector) {
            
            float ratioX = detector.getFocusX() / displayMetrics.widthPixels;
            float ratioY = detector.getFocusY() / displayMetrics.heightPixels;

            Log.d(TAG, "onScale: Called X: " + ratioX + ", Y: " + ratioY);

            Camera camera = sceneView.getScene().getCamera();
            //camera.setWorldPosition(); <--- How to calculate vector3

            return false;
        }

        @Override
        public boolean onScaleBegin(ScaleGestureDetector detector) {
            isScaling = true;
            return true;
        }

        @Override
        public void onScaleEnd(ScaleGestureDetector detector) {
            isScaling = false;

        }
    }

Edit:

Essentially I'm trying to get the camera to zoom in/out where I'm pinching on the screen.

Is this not possible?

Answer 1

In order to convert a 2d point into a 3d point, the depth needs to be added. This is done by creating a Ray from the camera through the point. To get the Vector3, a distance along that ray is picked.

For example, in DrawingActivity, the screen tap is used to create the ray, then a Vector3 is created DRAW_DISTANCE from the camera.

MotionEvent tap;
Camera camera = fragment.getArSceneView().getScene().getCamera();
Ray ray = camera.screenPointToRay(tap.getX(), tap.getY());
Vector3 drawPoint = ray.getPoint(DRAW_DISTANCE);

You'll need to decide on a value for DRAW_DISTANCE based on your specific use case. (DRAW_DISTANCE will get smaller as you zoom).

You can look at ScaleGestureDetector to handle the pinch and stretch gestures.