Search code examples
c#unity-game-enginehololensmrtkwindows-mixed-reality

How can I add basic near interaction to HoloLens 1 using MRTK?

Possibly related to How can I simulate hand rays on HoloLens 1?

I want to use HoloLens 1 devices to simulate basic near interactions as provided by HoloLens 2.

Specifically, how can I perform the following mappings:

  1. Use hand position during "Ready" gesture to control PokePointer?
  2. Use hand position during "Tap-and-hold" gesture to control GrabPointer?

Since HL1 does not track hand orientation, I expect these need to be estimated similar to the example with hand rays.

I have tried creating a custom pointer per the answer above, and it works for hand rays but not for poke/grab as far as I can tell.

I've also created a custom poke pointer according to the example for WMR controllers at How to mimic HoloLens 2 hand tracking wIth Windows Mixed Reality controllers [MRTK2]?, and assigned it to the GGV controller in the same fashion, but somehow the hands don't seem to get detected for poke (or grab), only for hand rays.

(I'm using the Grab pose since HL1 does not seem to return index finger pose during Ready gesture, and since pointer pose seems to refer to the gaze pointer for HL1)

Solution

  • Ok,

    In case someone else is trying to get near interactions on HoloLens 1, this is how I got it working in the end:

    1. Create a custom input profile
    2. Based on PokePointer, create a custom poke pointer component for the GGV (Gaze-Gesture-Voice) Controller of HL1 with the following modifications:
      • use the (grip) Position from the base controller component instead of gaze position.
      • calculate the Rotation from Position (interpolate using head position as in the hand ray example)
      • updateEnabled toggle set to not check for hand enabled since GGV always returns false during Ready
      • make sure to inherit from PokePointer (needed for event handlers that only allow near interactions from PokePointer or derived classes)
    3. Create a custom pointer prefab that uses the custom pointer component.
    4. Update the pointer section to use the custom pointer
    5. Modify buttons to only require proximity, and not require pushing from the front since the push direction is not working/unreliable on HoloLens 1