ARKit - Projection of ARAnchor to 2D space

Question

I am trying to project an ARAnchor to the 2D space but I am facing on an orientation issue...

Below my function to project the top left, top right, bottom left, bottom right corner position to 2D space:

/// Returns the projection of an `ARImageAnchor` from the 3D world space
/// detected by ARKit into the 2D space of a view rendering the scene.
///
/// - Parameter from: An Anchor instance for projecting.
/// - Returns: An optional `CGRect` corresponding on `ARImageAnchor` projection.
internal func projection(from anchor: ARImageAnchor,
                         alignment: ARPlaneAnchor.Alignment,
                         debug: Bool = false) -> CGRect? {
    guard let camera = session.currentFrame?.camera else {
        return nil
    }

    let refImg = anchor.referenceImage
    let anchor3DPoint = anchor.transform.columns.3

    let size = view.bounds.size
    let width = Float(refImg.physicalSize.width / 2)
    let height = Float(refImg.physicalSize.height / 2)

    /// Upper left corner point
    let projection = ProjectionHelper.projection(from: anchor3DPoint,
                                              width: width,
                                              height: height,
                                              focusAlignment: alignment)
    let topLeft = projection.0
    let topLeftProjected = camera.projectPoint(topLeft,
                                      orientation: .portrait,
                                      viewportSize: size)

    let topRight:simd_float3 = projection.1
    let topRightProjected = camera.projectPoint(topRight,
                                       orientation: .portrait,
                                       viewportSize: size)

    let bottomLeft = projection.2
    let bottomLeftProjected = camera.projectPoint(bottomLeft,
                                         orientation: .portrait,
                                         viewportSize: size)

    let bottomRight = projection.3
    let bottomRightProjected = camera.projectPoint(bottomRight,
                                          orientation: .portrait,
                                          viewportSize: size)

    let result = CGRect(origin: topLeftProjected,
                        size: CGSize(width: topRightProjected.distance(point: topLeftProjected),
                                     height: bottomRightProjected.distance(point: bottomLeftProjected)))

    return result
}

This function works pretty well when I am in front of the world origin. However, if I move left or right the calculation of the corner points does not work.

Answer 1

I found a solution to get corner 3D points of an ARImageAnchor depending on the anchor.transform and project them to 2D space:

    extension simd_float4 { 
        var vector_float3: vector_float3 { return simd_float3([x, y, z]) } 
    }

    /// Returns the projection of an `ARImageAnchor` from the 3D world space
    /// detected by ARKit into the 2D space of a view rendering the scene.
    ///
    /// - Parameter from: An Anchor instance for projecting.
    /// - Returns: An optional `CGRect` corresponding on `ARImageAnchor` projection.
    internal func projection(from anchor: ARImageAnchor) -> CGRect? {
        guard let camera = session.currentFrame?.camera else {
            return nil
        }
        
        let refImg = anchor.referenceImage
        let transform = anchor.transform.transpose

        
        let size = view.bounds.size
        let width = Float(refImg.physicalSize.width / 2)
        let height = Float(refImg.physicalSize.height / 2)
        
        // Get corner 3D points
        let pointsWorldSpace = [
            matrix_multiply(simd_float4([width, 0, -height, 1]), transform).vector_float3, // top right
            matrix_multiply(simd_float4([width, 0, height, 1]), transform).vector_float3, // bottom right
            matrix_multiply(simd_float4([-width, 0, -height, 1]), transform).vector_float3, // bottom left
            matrix_multiply(simd_float4([-width, 0, height, 1]), transform).vector_float3 // top left
        ]
        
        // Project 3D point to 2D space
        let pointsViewportSpace = pointsWorldSpace.map { (point) -> CGPoint in
            return camera.projectPoint(
                point,
                orientation: .portrait,
                viewportSize: size
            )
        }
        
        // Create a rectangle shape of the projection
        // to calculate the Intersection Over Union of other `ARImageAnchor`
        let result = CGRect(
           origin: pointsViewportSpace[3],
           size: CGSize(
               width: pointsViewportSpace[0].distance(point: pointsViewportSpace[3]),
               height: pointsViewportSpace[1].distance(point: pointsViewportSpace[2])
           )
        )
        
        
        return result
    }