I'm trying to store large, typed-arrays from Javascript into JSON-Files.
After stringifying one typed-array, I stored the JSON string in a new File using nodes' file stream system.
So far so good, but, once I checked the newly created file, I noticed that the size of this file is huge. After looking through the actual text stored in the file I noticed quite a lot of potential for improving for reducing the files size.
This exemplaric structure illustrates the structure of the array in the actual JSON string:
{"data": [{"my_key": "my_val1"},{"my_key": "my_val2"},{"my_key": "my_val3"},{"my_key": "my_val4"},{"my_key": "my_val5"},{"my_key": "my_val6"},{"my_key": "my_val7"},{"my_key": "my_val8"},{"my_key": "my_val9"}]}
Now, after googeling around a bit, I found this libary (json-compressed) which is exactly what I want to do with my array. Unfortunately this libary is made for php, I tried other libaries like JSONC and various other libaries that used some sort of compression algorithm. (LZW, DEFLATE/gzip).
I found that all libaries that use some sort of compression algorithm aren't able to provide a solution for my problem as I have to remove the keys before trying to compress the file even further.
Unfortunately, promising libaries like JSONC simply crashed my Application without throwing an error, notification or something else whenever I used their function JSONC.compress(). Their JSONC.pack() function crashes as well, but with giving away an error message. After looking through the issues related to JSONC on github I realized that this project is probably dead anyways.
Now back to my original question, is it possible to remove the keys from each value in the array? And if it is: Whats the smartest approach to do so?
To clear things up, the keys are NOT CREATED by me or PRESENT in my object! I simply want to stringify a TYPED-ARRAY but without getting a key for each value inside this array in the final JSON-STRING.
A result like this would be perfect and decrease the size of my file tremendously:
{"data": {"keys": ["my_key"], "data":["my_val1","my_val2","my_val3","my_val4","my_val5","my_val6","my_val7","my_val8","my_val9"]}}
JSON.stringify treats typed arrays as objects with 0-based indices rather than arrays. Converting a typed array to a standard array using Array.from before encoding would work, for example:
var myTypedArray = new Float32Array([0.1, 323.3, 0, 2.2]);
console.log(JSON.stringify(myTypedArray)); // looks like an object
var stdArray = Array.from(myTypedArray);
console.log(JSON.stringify(stdArray)); // looks like an array
Then to convert the JSON back to a typed array:
var json = "[0.1, 323.3, 0, 2.2]";
var myTypedArray = new Float32Array(JSON.parse(json));