Efficiently updating ARSCNFaceGeometry from a set of blend shapes

Question

I am using ARSCNFaceGeometry and need to update the face model's blend shapes as part of my game loop. My current solution is to call ARSCNFaceGeometry.update with a new ARFaceGeometry:

class Face {
    let geometry: ARSCNFaceGeometry
    
    init(device: MTLDevice) {
        guard let geometry = ARSCNFaceGeometry(device: device, fillMesh: true) else {
            fatalError("Could not create ARSCNFaceGeometry")
        }
        self.geometry = geometry
    }
    
    func update(blendShapes: [ARFaceAnchor.BlendShapeLocation: NSNumber]) {
        let faceGeometry = ARFaceGeometry(blendShapes: blendShapes)!
        geometry.update(from: faceGeometry)
    }
}

However this is not suitable for realtime usage since the ARFaceGeometry line alone takes around 0.01 seconds (just for reference, at 60fps we have a total frame budget of 0.0166 seconds).

After a few hundred updates or so, we seem to run to some sort of bug that takes the entire game loop to 10-20fps. Here's a 6 second sample from the profiler that sees ARFaceGeometry taking 2.3 seconds!:

Is there a more efficient way to update an existing ARSCNFaceGeometry from a set of blend shapes?

With custom 3D models for example, I can just update the blend shape values on SCNNode.morpher. Is there an equivalent for ARSCNFaceGeometry?

Answer 1

I could not track down the ARFaceGeometry performance problems. To workaround this, I instead decided to build my own model from ARFaceGeometry.

First I generated a model file from ARFaceGeometry. This file includes the base geometry as well as the geometry when each individual blend shape is applied:

let LIST_OF_BLEND_SHAPES: [ARFaceAnchor.BlendShapeLocation] = [
    .eyeBlinkLeft,
    .eyeLookDownLeft,
    // ... fill in rest
]

func printFaceModelJson() {
    // Get the geometry without any blend shapes applied 
    let base = ARFaceGeometry(blendShapes: [:])!
    
    // First print out a single copy of the indices.
    // These are shared between the models
    let indexList = base.triangleIndices.map({ "\($0)" }).joined(separator: ",")
    print("indexList: [\(indexList)]")
    
    // Then print the starting geometry (i.e. no blend shapes applied)
    printFaceNodeJson(blendShape: nil)
    
    // And print the model with each blend shape applied
    for blend in LIST_OF_BLEND_SHAPES {
        printFaceNodeJson(blendShape: blend)
    }
}

func printFaceNodeJson(
    blendShape: ARFaceAnchor.BlendShapeLocation?
) {
    let geometry = ARFaceGeometry(blendShapes: blendShape != nil ? [blendShape!: 1.0] : [:])!
    
    let verticies = geometry.vertices.flatMap({ v in [v[0], v[1], v[2]] })
    let vertexList = verticies.map({ "\($0)" }).joined(separator: ",")
    print("{ \"blendShape\": \(blendShape != nil ?  "\"" + blendShape!.rawValue + "\"" : "null"), \"verticies\": [\(vertexList)] }")
}

I ran this code offline to generate the model file (quickly converting the output by hand to proper json). You could also use a proper 3D model file format, which would likely result in a smaller model file.

Then for my app, I reconstruct the model from the json model file:

class ARMaskFaceModel {
    
    let node: SCNNode
    
    init() {
        let data = loadMaskJsonDataFromFile() // implement this!
        
        let elements = [SCNGeometryElement(indices: data.indicies, primitiveType: .triangles)]
        
        // Create the base geometry
        let baseGeometryData = data.blends[0]
        let geometry = SCNGeometry(sources: [
            SCNGeometrySource(vertices: baseGeometryData.verticies)
        ], elements: elements)
        
        node = SCNNode(geometry: geometry)
        
        // Then load each of the blend shape geometries into a morpher
        let morpher = SCNMorpher()
        morpher.targets = data.blends.dropFirst().map({ x in
            SCNGeometry(sources: [
                SCNGeometrySource(vertices: x.verticies)
            ], elements: elements)
        })
        node.morpher = morpher
    }

    /// Apply blend shapes to the model
    func update(blendShapes: [ARFaceAnchor.BlendShapeLocation : NSNumber]) {
        var i = 0
        for blendShape in LIST_OF_BLEND_SHAPES {
            if i > node.morpher?.targets.count ?? 0 {
                return
            }
            node.morpher?.setWeight(CGFloat(truncating: blendShapes[blendShape] ?? 0.0), forTargetAt: i)
            i += 1
        }
    }
}

Not ideal, but it works ok and performs much better, even without any optimization. We're pretty consistently at 60fps now. Plus it works on older phones too! (although printFaceModelJson must be run on a phone that supports realtime face tracking)