I am loading remote assets over XHR and receive them as arrayBuffers. The underlying data is float32 dependent. I need to do some further operations on these array buffers like concatenation and merging. For these operations I was experimenting with the buffer module.
I found that using the Buffer module and the underlying Uint8Array was changing the expected float32 output
When instantiating the TypedArray with the new Float32Array constructor, the data is correct.
new Float32Array(action.payload.arrayBuffer)
Float32Array(9072) [-5, 35, -5, 255, 127, 128, 128, 1, 0.2666666805744171, 0.2666666805744171, 0.2666666805744171, 1, -5, 34, -5, 255, 127, 128, 128, 1, 0.2666666805744171, 0.2666666805744171, 0.2666666805744171, 1, -5, 35, -2, 255, 127, 128, 128, 1, 0.2666666805744171, 0.2666666805744171, 0.2666666805744171, 1, -5, 34, -2, 255, 127, 128, 128, 1, 0.2666666805744171, 0.2666666805744171, 0.2666666805744171, 1, -5, 35, -2, 255, 127, 128, 128, 1, 0.2666666805744171, 0.2666666805744171, 0.2666666805744171, 1, -5, 34, -5, 255, 127, 128, 128, 1, 0.2666666805744171, 0.2666666805744171, 0.2666666805744171, 1, -5, 38, -2, 255, 127, 128, 128, 2, 0.9411764740943909, 0.3686274588108063, 0.10980392247438431, 1, -5, 34, -2, 255, 127, 128, 128, 2, 0.9411764740943909, 0.3686274588108063, 0.10980392247438431, 1, -5, 38, 1, 255, …]
When initializing the TypedArray with the from method the data is empty. This is expected because the from method expects an iterable array, which arrayBuffer is not.
Float32Array.from(action.payload.arrayBuffer)
Float32Array []
When converting from a Uint8Array or a Buffer which uses an underlying Uint8Array the float data is mutated.
Float32Array.from(new Uint8Array(action.payload.arrayBuffer))
Float32Array(36288) [0, 0, 160, 192, 0, 0, 12, 66, 0, 0, 160, 192, 0, 0, 127, 67, 0, 0, 254, 66, 0, 0, 0, 67, 0, 0, 0, 67, 0, 0, 128, 63, 137, 136, 136, 62, 137, 136, 136, 62, 137, 136, 136, 62, 0, 0, 128, 63, 0, 0, 160, 192, 0, 0, 8, 66, 0, 0, 160, 192, 0, 0, 127, 67, 0, 0, 254, 66, 0, 0, 0, 67, 0, 0, 0, 67, 0, 0, 128, 63, 137, 136, 136, 62, 137, 136, 136, 62, 137, 136, 136, 62, 0, 0, 128, 63, 0, 0, 160, 192, …]
Float32Array.from(Buffer.from(action.payload.arrayBuffer))
Float32Array(36288) [0, 0, 160, 192, 0, 0, 12, 66, 0, 0, 160, 192, 0, 0, 127, 67, 0, 0, 254, 66, 0, 0, 0, 67, 0, 0, 0, 67, 0, 0, 128, 63, 137, 136, 136, 62, 137, 136, 136, 62, 137, 136, 136, 62, 0, 0, 128, 63, 0, 0, 160, 192, 0, 0, 8, 66, 0, 0, 160, 192, 0, 0, 127, 67, 0, 0, 254, 66, 0, 0, 0, 67, 0, 0, 0, 67, 0, 0, 128, 63, 137, 136, 136, 62, 137, 136, 136, 62, 137, 136, 136, 62, 0, 0, 128, 63, 0, 0, 160, 192, …]
This difference in final output completely breaks my working pipeline that expects the new Float32Array output
I'm looking for an explanation and educational response on why this happens.
This is because every single value the Uint8 array (view) holds is cast to a full Float32 value based on the view, not the underlying ArrayBuffer.
If you have a Uint8Array view that holds for example 4 entries:
Uint8 -> [1,2,3,4]
In the ArrayBuffer it will look something like this (ArrayBuffer is always a byte array):
0x01020304
What happens then is that when you convert the Uint8 view to Float32 view, the values in the view representation (i.e. [1,2,3,4]) is used, not the underlying ArrayBuffer, so those entries are simply converted to new entries but in a different type that still represent the same numbers [1,2,3,4] as the original view represented:
Float32 -> [1,2,3,4] (each entry stored as 32-bit value)
And the new ArrayBuffer will look like (manually converted, but you'll get the idea) consisting of 4 bytes (32-bit) x 4 values the Float32 represents - also notice that floating point values are encoded in IEEE754 format using 4 bytes for each number a Float32 type represents:
0x00003F80 00000040 00004040 00004080 (=1,2,3,4 as IEEE754 encoded floating point values)
You can also see this from the byte size of the new underlying ArrayBuffer (from the numbers in the question):
9,072 x 4 = 36,288 bytes
In order to get the result you'd expect you have to use the buffer directly via its buffer property:
new Float32Array(uint8array.buffer); // notice .buffer -> actual ArrayBuffer
This is by the way the same as you do in the first line in the question:
new Float32Array(action.payload.arrayBuffer);
Byte-order is one aspect here but since your numbers when used directly from XHR with a Float32 view is correct, this is probably not an issue in this case.
So in short: there is no mutation taking place, but different types (via views) require different numbers of bytes as in this case, Uint8 a single byte from the underlying ArrayBuffer and Float32 4 bytes from the same, which means they will convert to different values as seem via the view.