Hardcoding large amounts of data without inducing memory spike

Question

So I am exploring the concept of storing WebAssembly inside the JavaScript file so it can all be bundled up in one shippable file. I did manage to make a working example of this where it stores the wasm file in a big literal string in base64 and at runtime is converted to a Uint8Array before being processed into a Module and Instance.

await Deno.writeTextFile(
    './static/wasm/bundle.js',
    `import { initSync } from './app.js'\ninitSync(new WebAssembly.Module(Uint8Array.from(atob('${btoa(
        [ ...await Deno.readFile('./static/wasm/app_bg.wasm') ]
            .map(byte => String.fromCharCode(byte))
            .join('')
    )}').split('').map(char => char.charCodeAt(0)))))`
)

Source of above snippet

But I have been wondering if JavaScript might have problems with processing this literal string in instances that the wasm file was very large. In this snippet the base64 literal string is only needed once at the very start, and I imagine is disposed off by the garbage collector as it's no longer accessible.

I am wondering if people have any ideas on how one could store this same type of data, hardcoded in the javascript, where it is only run once, but won't causes any huge memory spikes at the start of the runtime. Increased processing time for reduced peak memory usage is an acceptable trade-off here, but fetching any external resources would defeat the point of the question.

Answer 1

The following will reduce memory usage and load faster (CPU wise), but will result in a bigger file.

function toUint8ArrayString(u8) {
    return `new Uint8Array([${u8.join(',')}])`;
}

const wasmData = toUint8ArrayString(await Deno.readFile('./app_bg.wasm'));

await Deno.writeTextFile('./static/wasm/bundle.js', `import { initSync } from 'app.js'\ninitSync(new WebAssembly.Module(${wasmData}))`)

Instead of having the data in base64 you generate the array data to be used directly in the Uint8Array constructor.

From:

const init = atob('YQ==').split('').map(char => char.charCodeAt(0)); // [97]
Uint8Array.from(init);

To

new Uint8Array([97]); // or Uint8Array.from([97]);

By doing this, you're avoiding atob, .split, .map, .join with all the copies being done by these methods under the hood.

---

For large amounts of data, the best would be to use fetch and base64 encoded data:

const url = "data:application/wasm;base64," + b64wasm;
// Your bundle should produce the final url string
// const url = "data:application/wasm;base64,YQo=";
const res = await fetch(url);
const u8wasm = new Uint8Array(await res.arrayBuffer());
const module = new WebAssembly.Module(u8wasm);

And if supported, you can even use WebAssembly.compileStreaming which should result in the lowest amount of memory usage.

const url = "data:application/wasm;base64," + b64wasm;
// Your bundle should produce the final url string
// const url = "data:application/wasm;base64,YQo=";

const module = await WebAssembly.compileStreaming(fetch(url));