I'm trying to make an app, which would recognize a face from a webcam (Just return the landmarks of the face). I've written a code for a webcam and face analyze from a bitmap. But when I execute the code below, webcam just freezes. How I can use async/await to fix that? Additional question would be, how can I make my to call AnalyzeFace method only every 1 second? I don't really know how to do that yet, so I need your advices.
FaceDetectionFromFrame detects a face and draws a rectangle around it
form.scanPictureBox.Image shows current frame in a picture box
AnalyzeFace returns analyzed properties of a face
My frame processing code:
private static void ProcessFrame(object sender, EventArgs e)
{
List<string> faceList = new List<string>();
using (var imageFrame = capture.QueryFrame().ToImage<Bgr, Byte>())
{
FaceDetection.FaceDetectionFromFrame(imageFrame); // Face detection
var form = FormFaceDetection.Current;
form.scanPictureBox.Image = imageFrame.Bitmap;
faceList.Add(FaceRecognition.AnalyzeFace(imageFrame.Bitmap));
}
}
Although you didn't say so, it seems to me that `ProcessFrame is a function that is called whenever the camera wants to notify you that a new Frame is available for processing.
Apparently processing this new frame takes some considerable amount of time. It could even be that a new image is already grabbed when the previous one is not processed yet.
async-await won't help you with that: you'll have to grab the images as fast as possible and order another thread to process the fetched frame. You should return from the event handler as soon as possible, preferably before the frame has been processed. Another likely requirement is that the grabbed frames should be processed and displayed in the order that they were grabbed.
Below I'll tell you more about async-await. First I'll propose a solution for your camera problem.
The solution of your problem is in the producer-consumer design pattern. A producer produces data that has to be processed by a consumer. The data can be produced faster or slower than it can be processed by the consumer. If new data is available before the previously produced data has been processed by the consumer, the producer should save the produced data somewhere and continue producing.
Whenever the consumer has processed produced data it checks if there is more produced data and starts processing it.
This continues until the producer informs the consumer that no data will be produced anymore.
This produce-consumer pattern is implemented with all multi-threading safety in MSDN Task Parallel Library (TPL). This is downloadable as a nuget package: Microsoft Tpl Dataflow
You need two threads: a producer and a consumer. The producer produces images as fast as possible. Produced images are saved in a BufferBlock<Frame>.
A different thread will consume the produced image frames.
// the buffer to save frames that need to be processed:
private readonly BufferBlock<ImageFrame> buffer = new BufferBlock<ImageFrame>();
// event handler to be called whenever the camera has an image
// similar like your ProcessFrame
public async void OnImageAvailableAsync(object sender, EventArgs e)
{
// the sender is your camera who reports that an image can be grabbed:
ImageGrabber imageGrabber = (ImageGrabber)sender;
// grab the image:
ImageFrame grabbedFrame = imageGrabber.QueryFrame();
// save it on the buffer for processing:
await this.buffer.SendAsync(grabbedFrame);
// finished producing the image frame
}
The consumer:
// this task will process grabbed images that are in the buffer
// until there are no more images to process
public async Task ProcessGrabbedImagesAsync()
{
// wait for data in the buffer
// stop waiting if no data is expected anymore
while (await buffer.OutpubAvailableAsync())
{
// The producer put some data in the buffer.
// Fetch it and process it.
// This may take some time, which is no problem. If the producer has new frames
// they will be saved in the buffer
FaceDetection.FaceDetectionFromFrame(imageFrame); // Face detection
var form = FormFaceDetection.Current;
form.scanPictureBox.Image = imageFrame.Bitmap;
faceList.Add(FaceRecognition.AnalyzeFace(imageFrame.Bitmap));
}
}
Usage:
// Start a consumer task:
Task taskConsumer = task.Run( () => ProcessGrabbedImagesAsync());
// subscribe to the camera's event:
camera.EventImageAvailable += OnImageAvailableAsync;
camera.StartImageGrabbing();
// free to do other things
You'll need statements to Stop image grabbing
camera.StopImageGrabbing();
// unsubscribe:
camera.EventImageAvailable -= OnImageAvailableAsync;
// notify that no images will be produced:
buffer.Complete();
// await until the consumer is finished processing all produced images:
await taskConsumer;
async-await is only meaningful if your process has to wait idly for some other process to finish, for instance when it is waiting for a database query to finish, or a file to be written, or some information to be fetched from the internet. During this time your process normally would just wait idly until the other process is finished.
To make a function async:
Task instead of void and Task<TResult> instead of TResultTaskawait the returned Task<TResult> just before you need the result of the async function.await Task<TResult> is TResult; await Task has a void return.At first glance it seems that making your Your process would solve your problem.
private async void ProcessFrameAsync(object sender, EventArgs e)
{ // async event handlers return void instead of Task
var grabbedImage = await camera.FetchImageAsync();
// or if your camera has no async function:
await Task.Run( () => camera.FetchImage());
// this might be a length process:
ProcessImaged(grabbedImage);
ShowImage(grabbedImage);
}
If a new image is available before the previous is completely processed, then the event handler is called again. If the processing of the 2nd image is faster than the processing of the first image, then it is shown before the first image is shown.
Besides you'll have to take care that the two processes won't interfere with each other.
Hence in your case it is not a good idea to make an event handler async.
Only make event handlers async if you are certain that the event is completed before the next event is raised
If image-grabbing was not done via events, but directly, by asking the camera for a new image, async-await would help:
async Task GrabAndProcessImages(CancellationToken token)
{
// grab the first image:
var grabbedImage = await camera.GrabImageAsync(token);
while (!token.CancellationRequested)
{
// start grabbing the next image, do not wait for it yet
var taskGrabImage = camera.GrabImageAsync(token);
// because I'm not awaiting, I'm free to do other things
// like processing the last grabbed image:
ProcessImage(grabbedImage);
// await for the next image:
grabbedImage = await taskGrabImage;
}
}
Usage:
using(var cancellationTokenSource = new cancellationTokenSource())
{
Task taskProcessImages = grabAndProcessImages(cancellationTokenSource.Token);
// because I did not await, I'm free to do other things,
DoSomeThingElse();
// To stop grabbing images: cancel the cancellationTokenSource:
cancellationTokenSource.Cancel();
// or if you want to be sure that it grabbed for at least 30 seconds:
cancellationTokeSource.CanceAfter(TimeSpan.FromSeconds(30));
// still free to do something else,
DoSomeThingElse();
// before returning: await until the image grabbing task completes:
await taskProcessImages;
// if here, you are certain that processing images is completed
}