How to place a 3D model on top of a reference image with RealityKit?

Question

I want to place a 3D Model from the local device on top of the reference image when it's recognized. To achieve this I have tried the following:

1. Adding the reference image to the session configuration:

override func viewDidLoad() {
    super.viewDidLoad()
        
    arView.session.delegate = self
        
    // Check if the device supports the AR experience
    if (!ARConfiguration.isSupported) {
        TLogger.shared.error_objc("Device does not support Augmented Reality")
        return
    }
        
    guard let qrCodeReferenceImage = UIImage(named: "QRCode") else { return }
    let detectionImages: Set<ARReferenceImage> = convertToReferenceImages([qrCodeReferenceImage])
        
    let configuration = ARWorldTrackingConfiguration()
    configuration.detectionImages = detectionImages
        
    arView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
}

2. Using the ARSessionDelegate to get notified when the reference image was detected and placing the 3D model at the same position as his ARImageAnchor:

func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
    for anchor in anchors {
        guard let imageAnchor = anchor as? ARImageAnchor else { return }
        let position = imageAnchor.transform

        addEntity(self.localModelPath!, position)
    }
}

func addEntity(_ modelPath: URL, _ position: float4x4) {
    // Load 3D Object as Entity
    let entity = try! Entity.load(contentsOf: modelPath)
        
    // Create the Anchor which gets added to the AR Scene
    let anchor = AnchorEntity(world: position)
    anchor.addChild(entity)
    anchor.transform.matrix = position
    arView.scene.addAnchor(anchor)
}

However, whenever I try to place the anchor including my 3D model (the entity) at a specific position, it doesn't appear in the arView. It seems like the model is getting loaded though since a few frames are getting lost when executing the addEntity function. When I don't specifically set the anchors position the model appears in front of the camera.

Can anyone lead me in the right direction here?

Answer 1

Solution I

To make your code work properly, remove this line:

anchor.transform.matrix = position

@TimLangner – "...But I want to make the model appear on top of the reference image and not at any other location..."

As you can see, my test sphere has appeared on top of the reference image. When changing its position, remember that the Y-axis of the image anchor is directed towards the camera if the QR is vertical, and is directed up if the QR code is on a horizontal surface.

Make sure, a pivot point is located where it should.

In my case (you can see that I'm using AppClipCode), to move sphere 30 cm up, I have to move along negative Z direction.

entity.position.z = anchor.position.z - 0.3

Solution II

In my opinion, the most simple and productive solution would be to use the RealityKit's native AnchorEntity(.image(...)), without the need of implementing ARImageAnchor in delegate's method.

AnchorEntity(.image(group: "GroupName", name: "forModel"))

Here's a code:

import UIKit
import RealityKit

class ViewController: UIViewController {
    
    @IBOutlet var arView: ARView!
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        let entity = ModelEntity(mesh: .generateSphere(radius: 0.1))
        let anchor = AnchorEntity(.image(group: "AR Resources", 
                                          name: "appClipCode"))
        anchor.addChild(entity)
        
        entity.position.z = anchor.position.z - 0.3
        arView.scene.anchors.append(anchor)
    }
}