Search code examples
javascriptfirefoxlarge-files

Large blob file in Javascript

I have an XHR object that downloads 1GB file.

function getFile(callback)
{
    var xhr = new XMLHttpRequest();
    xhr.onload = function () {
        if (xhr.status == 200) {
            callback.apply(xhr);
        }else{
            console.log("Request error: " + xhr.statusText);
        }
    };

    xhr.open('GET', 'download', true);
    xhr.onprogress = updateProgress;
    xhr.responseType = "arraybuffer";
    xhr.send();
}

But the File API can't load all that into memory even from a worker it throws out of memory...

btn.addEventListener('click', function() {
    getFile(function() {
        var worker = new Worker("js/saving.worker.js");
        worker.onmessage = function(e) {
            saveAs(e.data); // FileSaver.js it creates URL from blob... but its too large
        };

        worker.postMessage(this.response);
    });
});

Web Worker

onmessage = function (e) {
    var view  = new DataView(e.data, 0);
    var file = new File([view], 'file.zip', {type: "application/zip"});
    postMessage('file');
};

I'm not trying to compress the file, this file is already compressed from server.

I thought storing it first on indexedDB but i i'll have to load blob or file anyway, even if i do request by range bytes, soon or late i will have to build this giant blob..

I want to create blob: url and send it to user after been downloaded by browser

I'll use FileSystem API for Google Chrome, but i want make something for firefox, i looked into File Handle Api but nothing...

Do i have to build an extension for firefox, in order to do the same thing as FileSystem does for google chrome?

Ubuntu 32 bits

Solution

  • I have a theory that is if you split the file into chunks and store them in the indexedDB and then later merge them together it will work

    A blob isn't made of data... it's more like pointers to where a file can be read from enter image description here

    Meaning if you store them in indexedDB and then do something like this (using FileSaver or alternative)

    finalBlob = new Blob([blob_A_fromDB, blob_B_fromDB])
saveAs(finalBlob, 'filename.zip')

    But i can't confirm this since i haven't tested it, would be good if someone else could