Search code examples
node.jsv8

In 2018 a Tech Lead at Google said they were working to "support buffers way beyond 4GiB" in V8 on 64 bit systems. Did that happen?

In 2018 a Tech Lead at Google said they were working to "support buffers way beyond 4GiB" in V8 on 64 bit systems. Did that happen?

Trying to load a large file into a buffer like:

const fileBuffer = fs.readFileSync(csvPath);

in Node v12.16.1 and getting the error:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (3461193224) is greater than possible Buffer: 2147483647 bytes.

and in Node v14.12.0 (latest) and getting the error:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (3461193224) is greater than 2 GB

Which looks to me to be a limit set due to 32 bit integers for addressing of the buffers. But I don't understand why this would be a limitation on 64 bit systems... Yes I realize I can use streams or read from the file at a specific address, but I have massive amounts of memory laying around, and I'm limited to 2147483647 bytes because Node is limited at 32 bit addressing?

Surely having a buffer of a high frequency random access data-set fully loaded into a buffer rather than streamed has performance benefits. The code involved in directing the request to pull from the multiple buffer alternative structure is going to cost something, regardless of how small...

I can use the --max-old-space-size=16000 flag to increase the maximum memory used by Node, but I suspect this is a hard-limit based on the architecture of V8. However I still have to ask since the tech lead at Google did claim they were increasing the maximum buffer size past 4GiB: Is there any way in 2020 to have a buffer beyond 2147483647 bytes in Node.js?

Edit, relevant tracker on the topic by Google, where apparently they were working on fixing this since at least last year: https://bugs.chromium.org/p/v8/issues/detail?id=4153

Solution

  • Did that happen?

    Yes, V8 supports very large (many gigabytes) ArrayBuffers nowadays.

    Is there any way to have a buffer beyond 2147483647 bytes in Node.js?

    Yes:

    $ node
Welcome to Node.js v14.12.0.
Type ".help" for more information.
> let b = Buffer.alloc(3461193224)
undefined
> b.length
3461193224

    That said, it appears that fs.readFileAsync has its own limit: https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L5 I have no idea what it would take to lift that. I suggest you file an issue on Node's bug tracker.

    FWIW, Buffer has yet another limit:

    > let buffer = require("buffer")
undefined
> buffer.kMaxLength
4294967295

    And again, that's Node's decision, not V8's.