I'm trying to send a large number of tasks to Azure Batch. There are three jobs and a total of 67,000 tasks spread across those jobs.
When I send only a few tasks everything works perfectly, as soon as I open it up to send the full list it seems to fall over, but without any useful error message. Is there some kind of limit on the total? Or the rate at which I can send them? I'm using Azure Functions, Node SDK and promises to add tasks asynchronously.
async.each(tasks, function(task, callback) {
var taskID = task.id + '_process';
var config = {
id: taskID,
displayName: 'Render portion ' + task.id + ' in job ' + task.job.id,
commandLine: "python3 main.py '" + JSON.stringify(task) + "'"
};
batchClient.task.add(task.job.id, config).then(function(result) {
context.log('Task for portion : ' + task.id + ' submitted successfully');
callback();
}).catch(function(error){
context.log('Task failed to be added');
callbacK(error);
});
}, function(error) {
if( error ) {
context.log('Error adding task.');
context.res = { body: error };
context.done();
} else {
context.log('All tasks have been queued successfully');
context.res = { body: 'All tasks have been queued successfully' };
context.done();
}
});
At this point, I'd be thrilled to even get an error, if only to give me something to go on. I've tried sending 10 tasks and it works fine, send 65,000 and it fails silently.
I suggest that you use the Add task collection API instead of the Add task API, as the collection API can take up to 100 tasks in a single API call. That should allow you to reduce the total number of API calls being made by a factor of 100.
If you use that API and are still having trouble, it's possible that there's some issue in the SDK with really high levels of parallelism... it might be worth trying to remove the call to Batch and confirm that there's no issue with the code you have as is (just replace it with a simple http get on a static URL maybe?)