I found a couple of questions pretty close to what I am looking for, but either I don't understand enough or the questions/answers do not exactly apply to what I am looking for:
Consider a very simple example of a requestAnimationFrame loop:
const someList = [];
function main() {
// Start the requestAnimationFrame loop
requestAnimationFrame(update)
}
function update() {
doSomething()
// Loop by calling update again with requestAnimationFrame
requestAnimationFrame(update);
}
function doSomething() {
someList.push(Math.floor(Math.random() * 10));
}
main();
Now here we do not have any problems. We start the loop, do something within the loop, call requestAnimationFrame again and everything is fine.
Now consider this next example. Here we have an async function called doSomethingAsynchronous and it has to be called with await. Therefore our update function needs to be async as well. My problem is that I do not understand how to work with requestAnimationFrame when the callback-function passed to the requestAnimationFrame is asynchronous:
function main() {
// Since update must now be async as well and called with await, what should this line look like?
requestAnimationFrame(update)
}
function async update() {
await doSomethingAsynchronous()
// Since update must now be async as well and called with await, what should this line look like?
requestAnimationFrame(update);
}
function async doSomethingAsynchronous() {
// Can't come up with an example, but this function should be called with await
}
main();
I feel like the requestAnimationFrame line in the code should probably look like one of these:
await requestAnimationFrame(update)
await requestAnimationFrame(async () => { await update(); });
requestAnimationFrame(async () => { await update(); });
But I'm not exactly sure if requestAnimationFrame can or should be awaited. In case I need to supply the update callback as an async function, that is also something I can't wrap my head around.
Edit:
In case someone is wondering why I am using requestAnimationFrame, the reason is that I am using TensorflowJS Object Detection model which returns the detection results asynchronously. So I want to asynchronously get the results and then render and visualize the results of a canvas - thus the requestAnimationFrame.
Edit 2:
As it has become apparent to me that this issue might not have a very generic solution, but instead the answer depends on the specific use case by a great deal, I'll add further additional information.
The issue is as follows:
predict function is asynchronous and it returns bounding boxes for the objects it detects in the video frameMy idea is to do all of this in a requestAnimationFrame loop. The reason why I need to perform an asynchronous function call within this loop is because the Object Detection Model's predict function is asynchronous.
So to put it simply:
Video ->
Pull frame ->
(*) Draw the frame on canvas ->
Asynchronously run Object Detection on the frame ->
Draw the Bounding Boxes returned by the Object Detection Model on the canvas (->)
(*) Draw the frame on canvas
(*) Preferably we should draw the video frame on canvas as soon as possible, before running the Object Detection, but I think it would be fine to do it after in case it is easier and/or less of a hassle. Either way we obviously do NOT want to draw the video frame twice, but instead only once at either point in time.
requestAnimationFrame (rAF) will queue your callback to be called right before the browser updates its rendering. This means that any new task¹ queued from such a callback will get executed only after the browser did update the rendering.
So having an async function inside an rAF callback is a sign of something potentially bad in your code design. If you did expect whatever data received asynchronously from that function to be rendered, it won't be in that frame. Worst, if you do prepare your rendering in different timings you may have half-baked frames, or even nothing at all, e.g if you clear a <canvas> in the synchronous part of your async function and then render whatever the data you received in the async function, at the time the browser renders, the <canvas> would be blank, and the painted frame would be cleared by the next frame call:
const canvas = document.querySelector("canvas");
const checkbox = document.querySelector("input");
const ctx = canvas.getContext("2d");
const getColor = async () => {
if (checkbox.checked) {
await new Promise((res) => setTimeout(res, 0));
}
return "green";
}
const asyncRenderer = async () => {
// prepare the context for rendering
ctx.fillStyle = "red";
ctx.fillRect(0, 0, canvas.width, canvas.height);
const color = await getColor();
ctx.fillStyle = color;
// won't be visible
ctx.fillRect(30, 30, 50, 50);
}
const loop = async () => {
while (true) {
await new Promise((res) => requestAnimationFrame(res));
await asyncRenderer();
}
};
loop();<label>await in getColor<input type=checkbox checked></label>
<canvas></canvas>Instead, you'll probably want to separate the updating phase of your animation and the rendering one.
How you do this will vary on the actual setup. Some may want to produce enough data as fast as possible so that they can buffer a few frames and be sure to be able to render every frames (e.g. in cases the data is generated from complex calculations in a Web Worker) and only produce more frames when reaching a threshold. Here the rendering loop could only trigger the data generator once in a while.
Some may prefer to require the data only when needed, e.g. if it comes from the network. In such a case, using rAF as a throttler is actually fine. You just have to be conscious that the data you request in rAF will be rendered one frame later, and that you may have some frames with no rendering.
And in your case, it seems that you're in a hybrid situation, where the data gathering is actually dependent on some external factor like the frequency at which new images are available to generate the object recognition on.
You don't say where your TensorFlow instance receives its input images from, but if we assume it's a video stream, then you will want to throttle the processing to both the stream frequency and the monitor frame-rate.
Assuming you gather the TensorFlow input from a <video> element, then you may want to have a look at this answer of mine where I describe various methods to extract every frames of a video, or this one where I explain how to get an event at each frame of a <video> element. Spoiler alert, it's not easy to get a cross-browser solution.
Once you have this running you'll need a more complex setup where you'll have one loop that awaits both that input event and rAF to gather new data, and another loop to render the new data when available, at the monitor's refresh rate.
// A simple helper
const waitForRAF = async () => {
return new Promise((res) => requestAnimationFrame(res));
}
const requestDataFromTensor = async () => {
// Send new data to the worker, wait for its response.
};
const waitForVideoInput = async () => {
// See https://stackoverflow.com/questions/32699721
}
let data = null; // We'll store the Tensor results in here
const updateDataFromTensorFlow = async () => {
await waitForVideoInput();
data = await requestDataFromTensor();
};
const updateLoop = async () => {
while (true) {
// Wait for both the data update and rAF,
// so we don't produce useless data in case
// it takes less than a monitor frame.
await Promise.all([
updateDataFromTensorFlow(),
waitForRAF(),
]);
}
}
const draw = (dataFromTensor) => {
// Actually draw on your <canvas>
}
const renderLoop = async () => {
while (true) {
await waitForRAF();
if (data) {
draw(data);
data = null; // Avoid redrawing if nothing new was received since last frame
}
}
}
// Now run both loops in parallel
updateLoop();
renderLoop();
Footnotes:
1. Also colloquially called "macrotask". Microtasks queued from an rAF callback will still execute before the next render, so having an async function in an rAF callback might work, if you ensure that the awaited data is actually already available, e.g. await Promise.race([getDataMaybeAlreadyThere(), Promise.resolve()]), but that is still a code-smell.