I am trying to build a downloader that automatically retries downloading. Basically, a task queue which retries tasks for a certain number of times. I first tried using Promise.all() but the "trick" to circumvent the fail-on-first-reject described here did not help (and is an anti-pattern as described further down in that thread)
So I got a version working which seems to somewhat do what I want. At least the results it prints are correct. But it still throws several uncaught exception test X errors/warnings and I don't know what to do about that.
The Code:
asd = async () => {
// Function simulating tasks which might fail.
function wait(ms, data) {
return new Promise( (resolve, reject) => setTimeout(() => {
if (Math.random() > 0.5){
resolve(data);
} else {
reject(data);
}
}, ms) );
}
let tasks = [];
const results = [];
// start the tasks
for ( let i = 0; i < 20; i++) {
const prom = wait(100 * i, 'test ' + i);
tasks.push([i, prom]);
}
// collect results and handle retries.
for ( let tries = 0; tries < 10; tries++){
failedTasks = [];
for ( let i = 0; i < tasks.length; i++) {
const task_idx = tasks[i][0];
// Wait for the task and check whether they failed or not.
// Any pointers on how to improve the readability of the next 6 lines appreciated.
await tasks[i][1].then(result => {
results.push([task_idx, result])
}).catch(err => {
const prom = wait(100 * task_idx, 'test ' + task_idx);
failedTasks.push([task_idx, prom])
});
}
// Retry the tasks which failed.
if (failedTasks.length === 0){
break;
} else {
tasks = failedTasks;
}
console.log('try ', tries);
}
console.log(results);
}
In the end, the results array contains (unless a task failed 10 times) all the results. But still uncaught exceptions fly around.
As not all rejected promises result in uncaught exceptions, my suspicion is, that starting the tasks first and applying then()/catch() later is causing some timing issues here.
Any improvements or better solutions to my problems are appreciated. E.g. my solution only allows retries "in waves". If anyone comes up with a better continuous solution, that would be much appreciated as well.
Using await and asnyc allows to solve that in a much clearer way.
You pass an array of tasks (functions that when executed start the given task) to the execute_tasks. This function will call for each of those tasks the execute_task, passing the task function to it, the execute_task will return a Promise containing the information if the task was successful or not.
The execute_task as a loop that loops until the async task was successful or the maximum number of retries reached.
Because each of the tasks has its own retry loop you can avoid those waves. Each task will queue itself for a new execution as it fails. Using await this way creates some kind of cooperative multitasking. And all errors are handled because the task is executed in a try catch block.
function wait(ms, data) {
return new Promise((resolve, reject) => setTimeout(() => {
if (Math.random() > 0.5) {
resolve(data);
} else {
reject(new Error());
}
}, ms));
}
async function execute_task(task) {
let result, lastError;
let i = 0
//loop until result was found or the retry count is larger then 10
while (!result && i < 10) {
try {
result = await task()
} catch (err) {
lastError = err
// maybe sleep/wait before retry
}
i++
}
if (result) {
return { success: true, data: result }
} else {
return { success: false, err: lastError }
}
}
async function execute_tasks(taskList) {
var taskPromises = taskList.map(task => execute_task(task))
// the result could be sorted into failed and not failed task before returning
return await Promise.all(taskPromises)
}
var taskList = []
for (let i = 0; i < 10; i++) {
taskList.push(() => {
return wait(500, {
foo: i
})
})
}
execute_tasks(taskList)
.then(result => {
console.dir(result)
})