ARRaycastQuery and Front Facing Camera (ARFaceTrackingConfiguration)

Question

I've searched all over the docs and might simply be missing something very big. Is raycasting available on front-facing True Depth cameras like on the iPad Pro?

I'm working on an application currently where I'm in ARFaceTrackingConfiguration and a simple raycast from the screen center is not yielding results.

That same code in World configuration using the rear camera is producing results. This code is straight from the docs.

let raycastQuery: ARRaycastQuery? = sceneView.raycastQuery(
                                                      from: sceneView.center,
                                                  allowing: .estimatedPlane,
                                                 alignment: .any)

let rayresults: [ARRaycastResult] = sceneView.session.raycast(raycastQuery!)
print(rayresults)

My understanding, given the examples around face tracking depth maps for background blur, was that the front camera would have essentially the same depth data as the rear, just with less total depth distance available.

Since this code returns a result set on the Rear Camera but does not return results on the Front Camera, I'm wondering if it is because of the direction of the ray? For the Front Camera, if rayCast is supported, maybe it requires the ray to be cast in the negative z direction? But I haven't had success there.

The goal of the project is to be able to get the distance between 2 points in 3D space but with the front camera not the rear.

Thanks for setting me straight! I appreciate it.

Answer 1

Hit-testing for ARFaceTrackingConfig

It's obvious that the ARKit raycasting methods used for detected planes based on World Tracking config will not work for the Face Tracking scenario. In your case, you need to use SceneKit's hit-testing. For hit-testing you will not be able to use estimatedPlane, you'll need a canonical face mask. And you are absolutely right about the XZ coordinates when using the front camera - the XZ plane is mirrored. It is also worth noting that the camera's IR TrueDepth sensor "works" at a distance from 10 to 100 cm. ARFaceAnchor is generated only within this range.

Run this code on your iPad Pro in the Swift Playgrounds app.

import ARKit
import SceneKit
import PlaygroundSupport

class ViewController : UIViewController {
    var sceneView = ARSCNView(frame: .zero)
    
    override func viewDidLoad() {
        super.viewDidLoad()
        self.view = self.sceneView
        sceneView.delegate = self
        sceneView.scene = SCNScene()
        
        let config = ARFaceTrackingConfiguration()
        config.maximumNumberOfTrackedFaces = 1
        sceneView.session.run(config)
    }
}

---

extension ViewController {       
    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        let location: CGPoint? = touches.first?.location(in: sceneView)
                                            
        var hitTestOptions = [SCNHitTestOption: Any]()
        hitTestOptions[.boundingBoxOnly] = true
                                            
        let hitTestResults: [SCNHitTestResult] = sceneView.hitTest(location!, 
                                                   options: hitTestOptions)
                    
        guard let hit = hitTestResults.first else { return }
                                        
        let sphere = SCNNode(geometry: SCNSphere(radius: 0.005))
        sphere.simdPosition = hit.simdWorldCoordinates
        sceneView.scene.rootNode.addChildNode(sphere)            
    }
}

---

extension ViewController : ARSCNViewDelegate {
    func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
        let faceGeo = ARSCNFaceGeometry(device: sceneView.device!)
        let node = SCNNode(geometry: faceGeo)
        node.geometry?.firstMaterial?.lightingModel = .physicallyBased
        node.geometry?.firstMaterial?.diffuse.contents = UIColor.green
        return node
    }
    func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode,
                                                    for anchor: ARAnchor) {    
        if let faceAnchor = anchor as? ARFaceAnchor,
           let faceGeo = node.geometry as? ARSCNFaceGeometry {
            faceGeo.update(from: faceAnchor.geometry)
        }
    }
}

PlaygroundPage.current.needsIndefiniteExecution = true
PlaygroundPage.current.liveView = ViewController()