Using Vuforia provided Projection Matrix and Marker Pose in SceneKit

Question

Currently I'm trying to troubleshoot the use of the projection matrix and framemarker pose when rendering in SceneKit. The model in the scene and the camera image background appear without problems. However, once I change the projection matrix and framemarker pose matrix to match Vuforia everything is pushed offscreen.

func didUpdateProjectionMatrix(projectionMatrix: matrix_float4x4)
{
    let extrinsic = SCNMatrix4FromMat4(projectionMatrix)
    let camera = self.cameraNode?.camera
    camera?.setProjectionTransform(extrinsic)
}

func didUpdateFramemarkers(framemarkers: [Framemarker]?)
{
    guard let framemarkers = framemarkers else {
        return
    }

    for framemarker in framemarkers {
        let pose = SCNMatrix4Invert(SCNMatrix4FromMat4(framemarker.pose))
        self.objectNode?.transform = pose
    }
}

Is this a correct way to setup the SCNCamera and object node and is there anything else required to setup Vuforia framemarkers to work with SceneKit?

Answer 1

It just works!

The hard part is determining what pieces of SceneKit are necessary to make this work. Originally I read the article Making Augmented Reality app easily with Scenekit + Vuforia which outlined how to rejigger the sample app for user-defined targets. The downsides to that article include that it isn't always clear what the author changed, no sample project is provided, and it is based upon an older version of Vuforia. Ultimately, I found it unnecessary to invert the pose matrix.

Draw camera image and set projection matrix and update marker pose

override func viewDidLoad() 
{
    super.viewDidLoad()

    let scene = SmartScanScene()

    let camera = SCNCamera()
    let cameraNode = SCNNode()
    cameraNode.camera = camera
    scene.rootNode.addChildNode(cameraNode)
    _cameraNode = cameraNode

    let view = self.view as! SCNView
    view.backgroundColor = UIColor.blackColor()
    view.showsStatistics = true
    // view.debugOptions = SCNDebugOptions.ShowBoundingBoxes.union(.ShowWireframe)
    view.autoenablesDefaultLighting = true
    view.allowsCameraControl = false
}

func didUpdateProjectionMatrix(projectionMatrix: matrix_float4x4)
{
    let extrinsic = SCNMatrix4FromMat4(projectionMatrix)
    _cameraNode?.camera?.setProjectionTransform(extrinsic)
}

func didUpdateFramemarkers(framemarkers: [Framemarker]?)
{
    guard let framemarkers = framemarkers else {
        return
    }

    for framemarker in framemarkers {
        let pose = SCNMatrix4FromMat4(framemarker.pose)
        self.objectNode?.transform = pose
    }
}

func didUpdateCameraImage(image: UIImage?)
{
    if let image = image {
        _scene?.background.contents = image
    }
}