Can't get FaceTracker class to work on HoloLens 2

Question

I'm having trouble getting the FaceTracker Class to work on HoloLens 2. As soon as I try to detect the faces with ProcessNextFrameAsync Method I get an exception of the following kind:

System.Runtime.InteropServices.COMException (0x80004005): Unspecified error

This is only the first part of the error message, if more information is needed, I can add that.

See this for a minimal example.

public async void Start()
{
    var selectedGroup = await FindCameraAsync();
    await StartMediaCaptureAsync(selectedGroup);
}

private async Task StartMediaCaptureAsync(MediaFrameSourceGroup sourceGroup)
{
    faceTracker = await FaceTracker.CreateAsync();
    this.mediaCapture = new MediaCapture();
    await this.mediaCapture.InitializeAsync(settings);
    this.frameProcessingTimer = ThreadPoolTimer.CreatePeriodicTimer(ProcessCurrentVideoFrameAsync, timerInterval);
}

private async Task ProcessCurrentVideoFrameAsync()
{
    const BitmapPixelFormat InputPixelFormat = BitmapPixelFormat.Nv12;
    var deviceController = this.mediaCapture.VideoDeviceController;
    this.videoProperties = deviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;
    VideoFrame videoFrame = new VideoFrame(InputPixelFormat, (int)this.videoProperties.Width (int)this.videoProperties.Height);

    IList<DetectedFace> detectedFaces;
    try
    {
        detectedFaces = await faceTracker.ProcessNextFrameAsync(videoFrame);
    }
    catch (Exception e)
    {
        System.Diagnostics.Debug.WriteLine($"Failed with Exception: {e.ToString()}");
        return;
    }

    videoFrame.Dispose();
}

1. i get a suitable camera with MediaFrameSourceKind.Color and MediaStreamType.VideoPreview with FindCameraAsync(). Which works fine in my opinion.
2. start MediaCapture and the FaceTracker within StartMediaCaptureAsync()
3. try to detect faces in ProcessCurrentVideoFrameAsync()

Here are the things I have tested and the information I have received:

• I have a picture in the format Nv12, PixelWidth 1504 and PixelHeigt 846
• the permissions in Unity are granted for Webcam, PicturesLibrary and Microphone
• the app is compiled with Il2CPP
• the message No capture devices are available. appears after starting the app. In other articles it was mentioned that the permission (Webcam or Microphone) is missing, which is not the case. But may be connected nonetheless.
• I used Track faces in a sequence of frames and Basic Face Tracking sample as reference

I am very grateful for every incentive and thought.

UPDATE 14. July 2020

I have just tried the FaceDetector on several individual images that were stored locally on the HoloLens 2. This works fine.

Even though FaceDetector and FaceTracker are not identical, they are very similar. So I guess that the problem is somehow related to MediaCapture.

Next I will try to capture an image with MediaCapture and process it with FaceDetector.

If anyone has any more ideas in the meantime, I would be grateful to hear them.

Answer 1

This is an official sample show how to use the FaceTracker class to find human faces within a video stream: Basic face tracking sample. And in line 256, that is the main point to get a preview frame from the capture device.

However, base on your code, you have created a VideoFrame object and specified the properties and format to it, but you are missing invoke GetPreviewFrameAsync to convert the native webcam frame into the VideoFrame object.

You can try the following code to fix it:

private async Task ProcessCurrentVideoFrameAsync()
{
    const BitmapPixelFormat InputPixelFormat = BitmapPixelFormat.Nv12;
    var deviceController = this.mediaCapture.VideoDeviceController;
    this.videoProperties = deviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;
    VideoFrame videoFrame = new VideoFrame(InputPixelFormat, (int)this.videoProperties.Width (int)this.videoProperties.Height);

//add this line code.
    await this.mediaCapture.GetPreviewFrameAsync(videoFrame);