Using RealityKit and SceneKit together

Question

I am trying to create an app where I can use the depth functionalities of RealityKit but the AR drawing capabilities from SceneKit. What I would like to do, is recognize an object and place a 3d model over it (which works already).

When that is completed I would like the user to be able to draw on top of that 3d model (which works fine with SceneKit, but makes the 3d model jitter). I found SCNLine to do the drawing, but since it uses SceneKit I can not use it in the ARView of RealityKit.

I have seen this already, but it does not cover fully what I would like.

Is it possible to use both?

Answer 1

SceneKit and RealityKit are incompatible due to a complete dissimilarity – difference in scenes' hierarchy, difference in renderer and physics engines, difference in component content. What's stopping you from using SceneKit + ARKit (ARSCNView class)?

ARKit 6.0 has a built-in Depth API (the same API is available in RealityKit) that uses a LiDAR scanner to more accurately determine distances in a surrounding environment, allowing us to use plane detection, raycasting and object occlusion more efficiently.

For that, use sceneReconstruction instance property and ARMeshAnchors.

import ARKit
import SceneKit

class ViewController: UIViewController {

    @IBOutlet var sceneView: ARSCNView!

    override func viewDidLoad() {
        super.viewDidLoad()

        sceneView.scene = SCNScene()
        sceneView.delegate = self

        let config = ARWorldTrackingConfiguration()
        config.sceneReconstruction = .mesh
        config.planeDetection = .horizontal
        sceneView.session.run(config)
    }
}

Delegate's method:

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, 
                                                 for anchor: ARAnchor) {

        guard let meshAnchor = anchor as? ARMeshAnchor else { return }
        let meshGeo = meshAnchor.geometry

        // logic ...

        node.addChildNode(someModel)
    }
}

P. S.

This post will be helpful for you.