How to read large files (Excel) in NodeJS without pausing UI

Question

I'm currently using ExcelJS to read large Excel files (10,000+ lines) into a NodeJS/Angular/Electron app. It reads the smaller files just fine, but the larger files take anywhere from 3.9 to 5 seconds to read, and during that time, the CSS doesn't update at all.

I'm currently using async/await to load the files, as I thought that would allow other operations to happen while waiting on the file load, as I thought I/O operations in Node had their own thread. I also read that CPU intensive tasks block all other Node processes.

This is the code that currently loads the workbook:

async openWorkbook(filename: string) {
    this.loading = true;
    const workbook = new Workbook();
    const promise = workbook.xlsx.readFile(this.path + '/sheets/' + filename)
      .then(() => {
        // use workbook
        workbook.getWorksheet(1).name = workbook.getWorksheet(1).name.slice(0, 31);
        const excelFile: ExcelFile = { workbook: workbook, filename: filename };
        this.wbSource.next(excelFile);
      });

    const read = await promise;
  }

This code works the same as the callback implementation, they both block the app's CSS. How would I go about reading a file (or do any CPU intensive task) without blocking the UI?

Answer 1

The first thing i would do in this case is preventing memory overhead by not "opening" large files at all.

So something like this should work:

const stream = fs.createReadStream(filePath);
const workbook = new Excel.Workbook();
stream.pipe(workbook.xlsx.createInputStream());

and as those chunks block the cycle for a minor amount of time you can use this even in the javascript event loop ;)