Detecting finger on AR object?

Question

I'm trying to create a mobile AR app to create a virtual piano keyboard on a flat surface and be able to play the keys using my fingers.

1) What tool would be the easiest way to achieve this? (e.g. Vuforia, ARKit)

2) Is it possible to show my fingers in front of the piano keyboard and not behind it?

Thank you so much for your help in advance.

Answer 1

What framework to choose for working with Occlusion :

Both frameworks ARKit and Vuforia support Occlusion feature but this feature isn't ideal at the moment. For me, the easiest way to implement occlusion in AR app is to use ARKit in Xcode. If you prefer Vuforia in Unity and its UI settings – you're welcome too.

How to use Occlusion in ARKit and Vuforia :

To composite your hand/finger over a real-world object you must use People Occlusion feature in ARKit or Occlusion Management in Vuforia.

It's extremely easy to activate Depth-channel compositing in ARKit using frameSemantics instance property – but remember that it works in iOS 13+ with A12 chip and higher:

let session = ARSession()

if let config = session.configuration as? ARWorldTrackingConfiguration {
    config.frameSemantics.insert(.personSegmentationWithDepth)
    session.run(config)
}

If you want to find out how to turn on Occlusion Management in Unity/Vuforia download a CylinderTargets that demonstrates cylindrical objects usage, along with the occlusion effect.

How to implement a touch :

To make a touch (when real finger touches AR Piano Keys) you must implement a collision. You have to activate Image Detection AI feature to detect where a finger's nail is, then place there an anchor, and tether to that anchor a small invisible sphere. You need to constantly update a position for that anchor at 60 fps. Then create a collision shape for piano key and a second collision shape for that sphere. Then allow them collide. If you're developing with ARKit use SceneKit framework to implement a collision event. If you're developing with Vuforia – use official documentation to find out how to do it.