I'm getting the following error even when I run node with high heap memory using the following command: node --max-old-space-size=8000 manipulateFiles.js
FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
1: 0x8dc510 node::Abort() [node]
2: 0x8dc55c [node]
3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xec7bf2 [node]
6: 0x102d7b2 v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
7: 0x1030946 v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [node]
8: 0x1030f69 v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::EnsureGrowable(v8::internal::Handle<v8::internal::OrderedHashMap>) [node]
9: 0x1114c7e v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x908bb15be1d
Aborted (core dumped)
The amount of heap memory used at the time of the crash is 1.79G. The amount available is 6.15G. I used the v8 module and process to get those numbers.
So apparently something else besides the heap size is causing the issue. the module basically scans a large CSV file and builds a Map for later reference in the process. This map can have up to 30 million keys. the module worked fine on other smaller files (10 million keys). but it keeps giving this error here even when the available heap size is far larger than being used.
What else could cause a problem like this?
V8 developer here. Despite the function names you see in the stack trace, this isn't about being out of memory overall, but about running into the maximum size for a single object (the Map in this case). See this answer explaining the details: https://stackoverflow.com/a/54466812/6036428