Search code examples
augmented-realityscenekitgltfrealitykitmodelio

How to load 3D content into RealityKit Apps at runtime?

For a specific purpose I have to load 3D content (glTF absolutely preferred) into my RealityKit based App during runtime. As far as I understand, RealityKit usually relies on 3D content conversion to proprietary .reality during compilation.

To make it more understandable: The app could be compared to IKEA Place app with changing AR content, or even the possibility to open 3D content attached to mails.

What is the state of the art solution to this? I found this approach: https://the-nerd.be/2014/11/07/dynamically-load-collada-files-in-scenekit-at-runtime/ But for my case it's not really usable as it relies on Mac-based precompiling of the 3D content and using SceneKit not RealityKit. I also heard of Model I/O (https://developer.apple.com/documentation/modelio) which might be a solution but my knowledge isn't sufficient for evaluating that. I'm in doubt if it's even usable for RealityKit or only SceneKit.

Please correct me if I'm mixing up SceneKit/RealityKit/ARKit.

Solution

  • RealityKit only supports loading USDZ files and projects specificaly made for RealityKit. RealityKit builds on top of ARKit, but uses Entities. Supported formats are:

    usdz, rcproject, reality

    let url = URL(fileURLWithPath: "path/to/MyEntity.usdz")
let entity = try? Entity.load(contentsOf: url)

    ARKit can load more file types using MDLAsset, but is also quite limited. It uses SCNNodes just like SceneKit. So everything you are able to load into SceneKit, can be used in ARKit. MDLAsset supports:

    abc, usd, usda, usdc, usdz, ply, obj, stl

    let url = URL(fileURLWithPath: "path/to/MyScene.usdz")
let asset = MDLAsset(url: url)
asset.loadTextures()
let scene = SCNScene(mdlAsset: asset)
return scene.rootNode.flattenedClone()

    There is a way how to load other types of models using AssimpKit - https://github.com/dmsurti/AssimpKit. You can load a scene and clone the root node (SCNNode), which can be then used in ARKit.

    AssimpKit is 3rd party library you can use to convert/import many file types into SceneKit during runtime. Supported types are:

    3d, 3ds, ac, b3d, bvh, cob, dae, dxf, ifc, irr, md2, md5mesh, md5anim, m3sd, nff, obj, off, mesh.xml, ply, q3o, q3s, raw, smd, stl, wrl, xgl, zgl, fbx, md3

    If supporting USDZ and RealityKit projects is not enough, you will probably need to use ARKit at the moment.