Draw over plane using hand in HoloLens2

Question

I want to implement a feature on HoloLens2 app, which allows user to draw/paint on specific surface in every part which was touched by the user.

So we have a flat plane and when I move my hand over this plane, spots below my hand should be coloured. How can I implement such feature keeping in mind, that HoloLens has limited processing power and calling Texture.Apply() every frame is unacceptable?

I tried to adopt a script from Eye tracking Heat map demo scene to use hand touch instead, but didn't success.

I've changed

        private void Start()
        {
            if (EyeTarget != null)
            {
                EyeTarget.WhileLookingAtTarget.AddListener(OnLookAt);
            }
        }

        public void OnLookAt()
        {
            if (UseLiveInputStream && (EyeTarget != null) && (EyeTarget.IsLookedAt))
            {
                DrawAtThisHitPos(EyeTrackingTarget.LookedAtPoint);
            }
        }

to

    public void OnTouchUpdated(HandTrackingInputEventData eventData)
    {
        DrawAtThisHitPos(eventData.InputData);
    }

and used IMixedRealityTouchHandler. But then HoloLens is extremely lagging and crashes

Any suggestions how to solve this problem?

Answer 1

HoloLens has limited processing power and calling Texture.Apply() every frame is unacceptable?

As say in Unity Documentation, Apply is a potentially expensive operation, so you'll want to change as many pixels as possible between Apply calls. In DemoVisualizer, the solution it uses is the dwellTimeInSec property provided by the EyeTrackingTarget component that defines the duration in seconds that the user needs to keep looking at the target to select it via dwell activation, and the duration value defaults to 0.8s. It avoids the overuse of the Apply method.

Therefore, you can refer to this practice of EyeTrackingTarget to improve the underperforming frame rate your mixed reality application got.