How to find the angle b/w 3d object and Vision Pro device camera( user view)

Question

I have a 3d object (billboard) which contains an ad on one side and I would like to identify whether it is visible (through vision device camera) or not for user.

Here is the code that I've used to calculate the angle, but seems to be it is not working as expected as I'm getting wrong angle value between camera and billboard. I have used dot product formula to do this. But not sure whether it is right way or not. Let me know what is the wrong in calculation or alternative way to do this.

class cityViewModel {
    var billboard = Entity()
}

struct CityView: View {
    private var cameraTracker = VisionProCameraTracker()
    var model = cityViewModel()
    
    var body: some View {
        RealityView { content in
            // Add the initial RealityKit content
            if let scene = try? await Entity(named: "newcity", in: citywithAdBundle) {
                content.add(scene)
                if let entity = scene.findEntity(named: "billboard") {
                    model.billboard = entity
                    print("Billboard position \(entity.position)")
                }
                updatingSceneEventsWith(content)
            }
        }
        .task {
            await cameraTracker.runArSession()
        }
    }
    
    private func updatingSceneEventsWith(_ content: RealityViewContent) {
        _ = content.subscribe(to: SceneEvents.Update.self) { _ in
            Task {
                let cameraTransform = await cameraTracker.getTransform()
                let x2 = cameraTransform!.columns.3.x
                let x1 = await model.billboard.position.x
                let y2 = cameraTransform!.columns.3.y
                let y1 = await model.billboard.position.y
                let z2 = cameraTransform!.columns.3.z
                let z1 = await model.billboard.position.z
                let distance = sqrtf( (x2-x1) * (x2-x1) +
                                      (y2-y1) * (y2-y1) +
                                      (z2-z1) * (z2-z1) )
                var formatted = String(format: "%.2f m", arguments: [distance])
                print("Distance from camera to billboard is:", formatted)
                
                let ab = (x1 * x2) + (y1 * y2) + (z1 * z2)
                let magnitudeOfAd = sqrtf( (x1 * x1) + (y1 * y1) + (y1 * y1) )
                let magnititeOfCamera = sqrtf( (x2 * x2) + (y2 * y2) + (y2 * y2) )
                let cosTheta = ab / (magnitudeOfAd * magnititeOfCamera)
                let angleInRadians = acos(cosTheta)
                let angleInDegrees = angleInRadians * 180 / .pi
                formatted = String(format: "%.2f m", arguments: [angleInDegrees])
                print("Angle b/w camera and billboard is \(formatted) degrees.")
            }
        }
    }
}

@Observable class VisionProCameraTracker {
    let session = ARKitSession()
    let worldTracking = WorldTrackingProvider()
    
    func runArSession() async {
        Task {
            try? await session.run([worldTracking])
        }
    }

    func getTransform() async -> simd_float4x4? {
        guard let deviceAnchor = worldTracking.queryDeviceAnchor(atTimestamp: 4)
        else { return nil }
        let transform = deviceAnchor.originFromAnchorTransform
        return transform
    }
}

Answer 1

Measuring the angle between Camera and Model

There's isNode(_:insideFrustumOf:) instance method in SceneKit, helping us to get a true value if the bounding box of the sought node intersects the perspective camera's frustum, defined by the POV node. Similar method would help us a lot. Unfortunately, RealityKit for visionOS 1.2 does not have such a useful method. So we'll be content only with what we have.

To solve your problem in the simplest way, I used the orientation(relativeTo:) instance method. This code allows you to find a 0...45 degree deflection angle along the Y and X axes in any direction. Set up the ARKit's Device Anchor tracking the same way as described in your question (since we'll use an object of the VisionProCameraTracker class). Then create an anchor entity and pass the transform matrix of the device anchor to it. Now you're capable of measuring the angle.

Here's my code:

import SwiftUI
import RealityKit
import ARKit

struct ContentView : View {
    let vpct = VisionProCameraTracker()
    let anchor = AnchorEntity(.head, trackingMode: .continuous)
    @State var billboard = Entity()
    
    var body: some View {
        RealityView { rvc in
            billboard = try! await Entity(named: "Billboard")
            rvc.add(billboard)
            rvc.add(anchor)
            
            let _ = rvc.subscribe(to: SceneEvents.Update.self) { _ in
                anchor.transform.matrix = vpct.getTransform()
                
                let radAngle = anchor.orientation(relativeTo: billboard).angle
                
                var radToDeg: Any = Int(radAngle * (180/Float.pi)) % 240
                
                if (radToDeg as! Int) > 45 {
                    radToDeg = "Angle is greater than 45 degrees"
                }
                if anchor.position.z < billboard.position.z {
                    radToDeg = "Billboard is not visible"
                }
                print(radToDeg)
            }
        }
        .task { await vpct.runArSession() }
    }
}

---

If you need to split the values of the X and Y angles, to discern not only a angle's magnitude but also its vector, use this code:

let _ = rvc.subscribe(to: SceneEvents.Update.self) { _ in
    anchor.transform.matrix = vpct.getTransform()
            
    let xVec = anchor.orientation(relativeTo: billboard).vector.x
    let yVec = anchor.orientation(relativeTo: billboard).vector.y
    let magnitude = anchor.orientation(relativeTo: billboard).angle.magnitude
               
    let radAngleX = xVec * magnitude  // radians
    let radAngleY = yVec * magnitude  // radians
            
    let radToDeg_X = Int(radAngleX * (180/Float.pi)) % 91
    let radToDeg_Y = Int(radAngleY * (180/Float.pi)) % 91

    print("x:", radToDeg_X, "y:", radToDeg_Y)
}

If you are interested in what the quaternion's real and imaginary parameters are, read this post.