How to get the mesh of an object in RealityKit from Experience.reality file

Question

I am loading a scene from a .reality file, and I want to give it an accurate collision because at the moment it is not accurate. So in order to do that, I want to generate a convex of the mesh of my object. The issue I'm having is that I don't find a way to retrieve the mesh of my object from a scene, tried to cast it to ModelEntity in order to get its .model property, but the casting fails for some reason.

guard let object = scene.objectName else { return } /// object is `Entity`
let model = object as! ModelEntity // Fails 

This is my code, I am looking for what should go into the generateConvex(from: ?) function:

Experience.loadSceneAsync { result in
  switch result:
  case let .success(scene):
      guard let object = scene.objectName else { return }
      object.components[CollisionComponent.self] = CollisionComponent(
                        shapes: [ShapeResource.generateConvex(from: ...)]
                    )
  case let .failure(error):
      ...
}

One approach I can think of, is to load the model separately as ModelEntity, then get its mesh, but it seems off.

How can I retrieve the mesh/geometry of my object from a scene, is there no way to get the mesh of an Entity class? is there something I should do in RealityComposer that I haven't done, and that's why the casting fails?

Answer 1

Apparently the model mesh transform is always the one the designer gave when creating it. Even though the Entity's transform might change, the mesh transform remains intact. Furthermore the bounds property on the mesh is a get only property, which means that it cannot be changed dynamically.

That said, there is a way to get a resized mesh in order to create a CollisionComponent with the correct size, by recreating the mesh, with the correct transform.

The steps are the following:

func collisionShape(modelName: String, onEntity entity: Entity) -> ShapeResource {
    // 1. Retrieve the current visual bounds of the entity.
    let entityBoundingBox = entity.visualBounds(relativeTo: nil)
        
    // 2. Locate the model entity among the children through the `ModelEntity`'s name,
    // although you can also do it recursively.
    let modelEntity = entity.findEntity(named: modelName) as! ModelEntity
    // 3. Grab the mesh.
    let modelEntityMesh = modelEntity.model!.mesh
        
    // 4. Calculate the scale on each dimension
    let xScale = entityBoundingBox.extents.x / modelEntityMesh.bounds.extents.x
    let yScale = entityBoundingBox.extents.y / modelEntityMesh.bounds.extents.y
    let zScale = entityBoundingBox.extents.z / modelEntityMesh.bounds.extents.z
        
    // 5. Create a new transform with the new scale.
    let scaledTransform = float4x4(scale: .init(x: xScale, y: yScale, z: zScale))
        
    // 6. Grab the models unchanged.
    let meshModels = modelEntityMesh.contents.models
    // 7. Grab the instances and replace their transform with the scaled one
    let transformedInstances = modelEntityMesh.contents.instances
        .map { instance -> MeshResource.Instance in
            var instance = instance
            instance.transform = scaledTransform
            return instance
        }
        
    // 8. Create the new `MeshResource.Contents`.
    var transformedContents = MeshResource.Contents()
    transformedContents.models = meshModels
    transformedContents.instances = MeshInstanceCollection(transformedInstances)
        
    // 9. Generate a new transformed `MeshResource` from the transformed contents.
    let transformedMesh = try! MeshResource.generate(from: transformedContents)
    let resource = ShapeResource.generateConvex(from: transformedMesh)
        
    return resource
}

With that you can now create the CollisionComponent you need:

let shapeResource = collisionShape(
    modelName: "MyModelName", 
    onEntity: MyRealityFileEntity
)
let collision = CollisionComponent(shapes: [shapeResource])

I use these convenience functions to get the scaled float4x4 for the transform:

extension float4x4 {

  init(scale factor: Float) {
        self.init(scale: SIMD3<Float>(repeating: factor))
    }
    
  init(scale vector: SIMD3<Float>) {
        self.init(SIMD4<Float>(vector.x, 0, 0, 0),
                  SIMD4<Float>(0, vector.y, 0, 0),
                  SIMD4<Float>(0, 0, vector.z, 0),
                  SIMD4<Float>(0, 0, 0, 1))
    }
}