I place a virtual model on the wall 10m away with RealityKit. I can not see the virtual model, though I can see the wall clearly. And when I enable debugoption.showSceneUnderstanding in RealityKit, the virtual model shows up. Or when I come closer to the wall, the virtual model can also show up. The configuration of ARView is as follows. Debug options are controlled by showMesh.
func makeUIView(context: Context) -> ARView {
let config = ARWorldTrackingConfiguration()
// Plane Detection
config.planeDetection = [.horizontal, .vertical]
// Environment Texturing
if #available(iOS 12, *) {
config.environmentTexturing = .automatic
}
// Person segmantantion
if (ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth)) {
config.frameSemantics.insert(.personSegmentationWithDepth)
config.frameSemantics.insert(.sceneDepth)
print("[Debug] People occlusion on")
} else {
print("[Debug] People occlusion not available on this devices")
}
// Use LiDAR to promote the scan ablity
if(ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh)){
config.sceneReconstruction = .mesh
print("[Debug] Scene reconstruction on")
} else {
print("[Debug] The device does not support LiDAR")
}
// Scene Understanding
arViewModel.arView.environment.sceneUnderstanding.options.insert(.occlusion)
arViewModel.arView.environment.sceneUnderstanding.options.insert(.receivesLighting)
// ARCoachingOverlay
arViewModel.arView.addCoaching()
// Debug
if showMesh {
arViewModel.arView.debugOptions.insert(.showAnchorOrigins)
arViewModel.arView.debugOptions.insert(.showSceneUnderstanding)
}
arViewModel.arView.session.run(config)
placementSetting.sceneObserver = arViewModel.arView.scene.subscribe(to: SceneEvents.Update.self, { (event) in
updateScene(for: arViewModel.arView)
})
return arViewModel.arView
}
Do I miss some configurations? Does ARKit support the visualization of objects at distance?
Here is the video telling what I met.
P.S.
The AR object disappears at 23 seconds and `debug option is enabled when the meshes of the scene appears at 40 seconds.
Update!: I find the problem is caused by:
arViewModel.arView.environment.sceneUnderstanding.options.insert(.occlusion)
When I disable the occulusion by removing the above sentence, the AR objects can be seen at distance but can not be occluded by real objects. Moreover, I find the problem may be irrelevant to lidar because I tried lidar+occlusion(disappear), lidar(work well without occlusion), occlusion(disappear), none(work well without occlusion).
Lidar is enabled by:
config.sceneReconstruction = .mesh
We know that LiDAR sensor on the iPad/iPhone does effectively work at a range of up to 5 meters. Literally its working distance is 0.5 to 4.9 m. RealityKit engineers have calculated that at this range, LiDAR will be able to reconstruct surfaces with a certain number of polygons. Therefore, it was decided to use dynamic tessellation of the reconstructed mesh to optimize processing.
Here's what NVidia docs tell us about dynamic tessellation:
Dynamic tessellation allows you to easily control the visual fidelity of the meshes in your scene on a per primitive basis as well as remove obnoxious mesh popping from LoD mesh swaps.The nature of the tessellation hardware allows you to specify a subdivision factor per edge and interior of each primitive dynamically. This is a powerful tool.
Using Scene Reconstruction feature without dynamic tessellation, in my opinion, is practically impossible in 5 nm iOS gadgets, because with static tessellation, the number of polygons in a 3D scene will quickly go over the border of 1 million. As you understand, today RealityKit isn't able to work with such a mesh resolution.
According to Apple documentation, AR scene must contain not greater than 100K polygons. If you follow this recommendation, your AR scene will be manageable and responsive. Also, it helps you save a battery life and play AR content without a drop frame.
Apple's documentation doesn't contain any specific information on using LiDARs for AR, but I think that technically the 10-meter-threshold for the reconstructed mesh can be enabled/disabled either by using the camera's far clipping plane parameter, or by registering the loss of the indexed triangular polygon that your Anchor stood on, or by multiplying RGBA by the farthest dark pixels of ZDepth channel. Whatever way is chosen...
I would like to say a few words about the Occlusion algorithm. In devices with a LiDAR scanner, information about the ZDepth channel is also collected through LiDAR – thus, if LiDAR collected incorrect info, we would get a low-quality ZDepth. In devices without a laser scanner, the ZDepth channel is generated using math applied to the Disparity.Left and Disparity.Right channels.
Regarding the occlusion feature in your case: since ZDepth channel is generated based on info coming from LiDAR scanner (but LiDAR, as we can see, runs into a 10-meter-constraint), it doesn't make much sense to conclude what feature works right and what wrong. There is definitely a distance-constraint in Scene Reconstruction feature at the moment. Therefore, you will not be able to effectively use Scene Reconstruction with Occlusion at distances over 10 meters.