Promise performance issue. Reading files too slow

Question

Im using promises to read 10000+ files in the hard drive and find a number at an exact position. Im using glob to return the filenames and for each file found i run the method readFile (readFile returns as promisse). When all files are processed i can continue my work.

function readFilesGlob(globPath,options,progressCallback){
    return new Promise(function (fulfill, reject){

      glob(globPath, options, function (err, files) {
        var readPromisses=[];
        for(var id in files){
          readPromisses.push(readFile(files[id]));
        }

        Promise.all(readPromisses).then(
          function(filesContents){
            fulfill(filesContents)
          }
        ); 

      });
    });
  }

all promises only complete when everything is done, making it unable to show progress of processing

function readFilesGlob(globPath,options,progressCallback){
    return new Promise(function (fulfill, reject){

      glob(globPath, options, function (err, files) {
        var readPromisses=[];
        for(var id in files){
          readFile(files[id]).then(function(data){
            //everything shows at the same time, like i was using the Promise.all
            console.log(data)
          })
        }
        //just testing the speed of the return in the console.log above
        fulfill();

      });
    });

The problem is. It is too slow and i only have a return several minutes later (or when i get out of memory)

Im thintking that im using promises wrong. Can someone give me an more performatic example to read a list of files with promises?

Answer 1

This seems like a excellent solution for the async package!

Checkout the each function: https://caolan.github.io/async/docs.html#each

Example:

// assuming openFiles is an array of file names
async.each(openFiles, function(file, callback) {

    // Perform operation on file here.
    console.log('Processing file ' + file);

    if( file.length > 32 ) {
      console.log('This file name is too long');
      callback('File name too long');
    } else {
      // Do work to process file here
      console.log('File processed');
      callback();
    }
}, function(err) {
    // if any of the file processing produced an error, err would equal that error
    if( err ) {
      // One of the iterations produced an error.
      // All processing will now stop.
      console.log('A file failed to process');
    } else {
      console.log('All files have been processed successfully');
    }
});

I use the async package in almost all my projects, it's fast and has loads of features <3

Alternative: Use a queue system

Setup a queue system, one I worked with is kue, this way you can also checkout the progress on your 'tasks'