Getting a UIImage or CIImage from ARReferenceImage

Question

I am using ARKit to do some animation over ARAnchors. My images are stored in an AR Resource Group like below.

As such they can be accessed like below in order to load them into my session as tracked images. However, I would also like to do some separate operations on these same images and therefore need to access them as either CIImages or UIImages. Right now I have to import these separately into my project because there doesn't seem to be a way to extract the image from an ARReferenceImage. Is there a way to do this? It is quite convenient to use the AR Resource Group as it displays relevant warnings about potential conflicts as well as less performant markers before compiling. I also assume it would take more time to load up the app if I were to convert say my PNG images into CIImages then into ARReferenceImage in my viewDidLoad() or by some similar means.

guard let trackedImages = ARReferenceImage.referenceImages(inGroupNamed: "Photos", bundle: Bundle.main) else {
    print("No images available")
    return
}

Answer 1

Creating ARReferenceImages in Xcode doesn’t put the original images into your app’s asset catalog, so the original image isn’t available at run time. (Instead, ARKit pre-processes the image into a form that makes recognition more efficient while using less storage.)

If you want to use an image both for ARKit recognition and for visual display (or other processing), you’ll need to include it in your Asset Catalog twice — once as an AR Reference Image, then again as a normal (UIImage) asset.

Alternatively, you could add it to your asset catalog as a normal image, then create an ARReferenceImage from it at run time — that class has an initializer that takes CGImage. (In that case, it might still be useful to use the AR Reference Images area of the asset catalog to preflight your images and make sure they’re good for recognition, even if you’re not delivering them that way.