I'm attempting to perform 3D feature detection using ARKit and SceneKit on an iPhone X, using Vision to detect facial landmarks. I then use ARSCNView.hitTest to cast the 2D face landmark against the face topology obtained with ARSCNFaceGeometry. This works great but I'm having trouble with the pupils. It's very hard to get hit test results with the pupils, and I'm assuming this is because the face geometry has gaps where the eyes are. Is there a way to get an accurate z-value from the leftPupil and rightPupil VNFaceLandmarkRegion2D coordinates by adding eyes to the face geometry to perform a hitTest, for example?
My hacky idea currently is to use the z values of the surrounding eye landmarks and use that value to unproject the 2D pupils into the scene, but of course this isn't as accurate as I would like it to be. Any help or any other ideas would be much appreciated, thank you!
If you pass fillMesh: true when you create the ARSCNFaceGeometry it won't leave an empty gap for the eyes and mouth.
https://developer.apple.com/documentation/arkit/arscnfacegeometry/2928202-init