Webrtc datachannels: saving data in file during transfer of big files

Question

I'm using the WebRTC data channels to build a file transfer service.

Its going quite good with smaller files, under 30 Mb or so. Right now on the receiving end I am simply saving the file data in memory, when all data is transferred I save the file.

Kinda like this :

//On the recieving side
var dataArray = [];
var dcOnMessage= function(event){
    dataArray .push(event.data);
    if(bytesToRecieve == 0)
    {
        var blob = new Blob(dataArray ,{type: incFileDesc.type});
        reader.onload = function (event) {
            saveToDisk(event.target.result,incFileDesc.name);
        }
        reader.readAsDataURL(blob);
    }
}

var saveToDisk = function(fileUrl, fileName) {
        var save = document.createElement('a');
        save.href = fileUrl;
        save.target = '_blank';
        save.download = fileName || fileUrl;
        var event = document.createEvent('Event');
        event.initEvent('click', true, true);

        save.dispatchEvent(event);
        (window.URL || window.webkitURL).revokeObjectURL(save.href);
    }

So I want to save the data on a file on disk, and then write directly to that file. But how do I do that?

Answer 1

I'm afraid the current standardized APIs don't easily allow that (see Philipp's response). The closest would be to save each as a blob/etc in localstorage/indexeddb, then use a Blob constructor to build the final file from the set of blobs. It will still have a temporary memory hit of roughly 2x filesize. Or just hold onto each blob in memory until building the final Blob and saving to disk (still a memory hit, but gradual until it goes 2x when building the final blob). These likely start having problems when the sizes of the final files get in the magnitude range of the available RAM.

Direct transfer of a single large Blob in Firefox->Firefox works today without SCTP ndata support (which isn't available yet) using a deprecated low-level chunking mechanism; it avoids the 2x part of the memory hit.

There's a non-standard API in Chrome that can mostly do the append-to-file part, last I checked. This has been an ongoing area of discussion with the WebAPI folk; it's probably time to poke them again.