Add UIImage as texture to a Plane in RealityKit

Question

I was trying out the new RealityKit framework in swift and I want to place a 2D image in on an ARAnchor. I managed to create a mesh plane and I tried to add the image as a texture to this plane. I found out that I can give a local image to a material. How can I use a UIImage instead?

let resource = try? TextureResource.load(named: "restart")
var material = UnlitMaterial()
material.baseColor = MaterialColorParameter.texture(resource!)
material.tintColor = UIColor.white.withAlphaComponent(0.99)

let imagePlane = ModelEntity(mesh: MeshResource.generatePlane(width: 0.25, height: 0.25), materials: [SimpleMaterial(color: color, isMetallic: true)])

Answer 1

Currently, you cannot use UIImage or CIImage as a shader's texture in RealityKit 2.0. In both versions of RealityKit, the texture must be loaded via the String type parameter of the load() method.

RealityKit 2.0

To assign a texture to a shader in RealityKit 2.0 use the following approach:

let mesh: MeshResource = .generatePlane(width: 0.45, depth: 0.45)
        
var material = SimpleMaterial()
material.color = .init(tint: .white.withAlphaComponent(0.999),
                    texture: .init(try! .load(named: "texture.png")))
material.metallic = .float(1.0)
material.roughness = .float(0.0)

let model = ModelEntity(mesh: mesh, materials: [material])

RealityKit 1.0

To assign a texture to a shader in RealityKit 1.0 use this approach:

let scene = try! Experience.loadMyScene()

var material = SimpleMaterial()
material.baseColor = try! .texture(.load(named: "texture.png"))
material.metallic = MaterialScalarParameter(floatLiteral: 1.0)
material.roughness = MaterialScalarParameter(floatLiteral: 0.0)
material.tintColor = UIColor.white

let mesh: MeshResource = .generatePlane(width: 0.45, depth: 0.45)
let component = ModelComponent(mesh: mesh, materials: [material])

scene.myFavoriteScene?.children[0].components.set(component)
arView.scene.anchors.append(scene)

CGImage

Nonetheless, you can create a texture resource from an in-memory Core Graphics image:

static func generate(from: CGImage, 
                 withName: String?, 
                  options: TextureResource.CreateOptions) -> TextureResource

Also, you can use a URL parameter:

material.color.texture = .init(try! .load(contentsOf: url))  // RealityKit 2.0