Create big json object js

Question

I am using Nodejs to Create a JSON file from a really large JSON object (1GB). In order to avoid memory problems, I'm using createWriteStream :

var writeStream = fs.createWriteStream('./output/outPut.json')

After using Builder(custom function ) it will return a big object JSON.

the final step is to create this file :

 writeStream.write(JSON.stringify(search_index), (err) => {
             if (err) throw err
             console.log('File  Ready... ')
                    })

but unfortunately, JSON.stringify cannot be used with such heavy Object

JSON.stringify throws RangeError: Invalid string length for huge objects

Is there a solution to this issue, please?

Answer 1

First step is to init your stream obejct

const writeStream = fs.createWriteStream('./object.json', { flags: 'w' })

then Converting my data to string JSON (stringify ) Using JSON Stream Stringify module

const JsonStreamStringify = require('json-stream-stringify')
const jsonStream = new JsonStreamStringify(Promise.resolve(Promise.resolve(TargetData)))

The last step is to pipe the returned data to writeStream

jsonStream.pipe(writeStream)
jsonStream.on('end', () => console.log('done '))

But It may take a lot of time in my case 10-15 min (1.1GB)