Reading large data from an xlsx using Javascript without sheet.js or spread.js

Question

The short version of the issue is that I need a way to read an xlsx file that contains 40k rows of 36 columns.

Shee.js runs out of memory, spread.js isn't an option due to our development environment, and I attempted to do a Rest call to get the document. Anything more than 300 rows just silently fails, and running the call in a loop to pull X rows repeatedly and combine the data takes nearly 4 minutes, which isn't really an option for a simple web based app.

We're in SharePoint. The point of the page is to provide a simplistic dashboard with data in an xlsx without having to import that data into a list.

Answer 1

The solution I've come to as of right now is to work asynchrounsly using $.ajax to pull individual columns.

I can pull back 10,000 items per ajax call, so I created an array of strings that contains the column names, did a forEach loop that runs a rest call to grab all of the data in that column (4 calls per column), then at the end of each loop i'm incrementing a number.

Once the number is equal to the column array -1 I'm stitching the columns into a single 2D array.

It's slower than I'd like, but it cuts the time from 4-9 minutes down to about 30 seconds. This is because instead of doing hundreds of calls in a chain, it's doing hundreds of calls simultaneously.

Edit: to expand on this answer for anyone that wants to use it in the future.

var cols = ['a', 'b', 'c', 'd', 'e', 'f'];
var data = {};
cols.forEach(function(item, I){
    data[item] = getData(item, 1, 100000);
});

Obviously not code complete and is just an example (I'm using a library I developed to fetch and format the data), but that's the basic idea. It will run an AJAX call on each column rather than the entirety of the document, keeping the call from failing.