How to project 3D Joint Points (Transformation) onto 2D Screen ARKit?

Question

I am trying to project the 3D human joint points onto the iPhone's screen using ARKit.

1. I am extracting the global transforms:

    let rightArmPosition = skeleton.modelTransform(for: ARSkeleton.JointName(rawValue: "right_arm_joint"))!
    let rootPosition = skeleton.modelTransform(for: .root)!

2. I am calculating the offset

    let rightOffset = simd_make_float3(rightArmPosition.columns.3)
    let rootOffset = simd_make_float3(rootPosition.columns.3)

3. I am projecting the points

let pMatrix = camera.projectionMatrix
        
let pRightOffset = camera.projectPoint(rightOffset, orientation: .portrait, viewportSize: CGSize(width: UIScreen.main.bounds.width, height: UIScreen.main.bounds.height))
        
let pRootOffset = camera.projectPoint(rootOffset, orientation: .portrait, viewportSize: CGSize(width: UIScreen.main.bounds.width, height: UIScreen.main.bounds.height))
        
humanJointsView.frame = UIScreen.main.bounds
humanJointsView.points = [pRightOffset, pRootOffset]

4. I am trying to draw the points in the target view:

override func draw(_ rect: CGRect) {
        path.removeAllPoints()
        self.points.forEach { point in
            path = UIBezierPath(ovalIn: CGRect(x: point.x, y: point.y, width: CGFloat(30), height: CGFloat(30)))
            UIColor.green.setFill()
            path.fill()
        }
    }

This approach is not working however, where is my mistake?

Thank you!

Answer 1

The right arm and root transforms are in skeleton model space, you need to multiply them with the body anchor transform.

rightArmWorldTransform = body.transform * rightArmPosition