Search code examples
iosaugmented-realityarkitwwdc

What is mean by Screen space in ARKit?


When I went through the video for session 805, Creating Great AR Experiences, and they mention using screen space text when displaying text in AR. They use the example of their Measure app that works quite well.

I want to achieve similar functionality. But, I am not able to figure it out.


Solution

  • Screen Space is the 2D pixel coordinate system of your display. It's where you normally do things like laying out UIKit widgets in screen space, etc.

    The alternative to Screen Space is a 3D coordinate space that relates to the 3D world tracking ARKit does or the 3D rendering engine you use to display virtual 3D content in AR. (In that case there are multiple 3D spaces to consider, like "absolute" world space vs camera-relative positioning.)

    As mentioned in the WWDC18 talk Creating Great AR Experiences, screen space is a great way to display content that's inherently 2D (like text) in a way that remains readable regardless of the distance to or orientation of a real-world feature that content is associated with. (If you were to place that text label in world space, it'd get small and unreadable as you get far away.)

    ARKit doesn't give you any automatic way to have 2D screen-space content follow a 3D world-space position, but making that happen yourself isn't especially difficult. Here's a general recipe for overlaying UIViews that follow 3D positions:

    1. Add an ARAnchor at the real-world position where you want screen-space content to appear.
    2. Use a per-frame callback (renderer(_:didUpdate:for:) or renderer(_:updateAtTime:) if you're using SceneKit with ARSCNView, session(_:didUpdate:) if you're building your own renderer) to get the current 3D position of that anchor.
    3. Project the 3D position (ARSCNView projectPoint, ARCamera projectPoint) to get the 2D point where that 3D position appears on screen.
    4. Set the position of your UIView to match that point.

    For more advanced cases like how the Measure app makes its screen-space label parallel to the line between two 3D points, you'd need to project the two endpoints and use a function like atan2 to get an angle for rotating the label.