Search code examples
jsonnode.jswebstorm

NodeJS JSON.parse(...) takes forever to finish (under debugger in WebStorm)

EDIT

The problem seems to be related to WebStorm itself, it seems that it doesn't want to work with objects containing huge amount of nested objects. Neither it wants to show the object contents inside Watches window. The problem is kinda strange because I'm able to inspect the string, it is loaded blazingly fast. Seems like a WebStorm issue

I have a relatively big JSON file 4.9mb that I need to process in NodeJS, the file is stored in file system and is loadded using following lines of code:

var path = require('path');

var filename = path.join(__dirname, 'db_asci.json');
var fs = require('fs');

var content = fs.readFileSync(filename);

debugger;
var decycledObj = JSON.parse(content);
debugger;

The problem is that after the first debugger; breakpoint is hit, the second one is not, I'm waiting for more than 20 minutes and nothing, one processor core is loadded at 100%. I'm unable to debug the function because it's native.

Here is ASCI version of JSON

Here is UTF8 version of JSON

What am I doing wrong?

Solution

  • The problem you are running in to is not JSON parsing taking too long. Indeed, try this:

    var start = Date.now();
var obj = JSON.parse(fs.readFileSync(filename));
console.log('Took', Date.now() - start, 'ms');

    You'll probably see that it took less than a second or so.

    What you are running into is an issue with the debugger itself – the observer effect. The act of observing a system changes that system.

    I assume you're using node-inspector. Whenever you have an extremely large, complex object, it is extremely expensive to load the object into the inspector. While it is doing so, your node process will peg the CPU and the event loop is paused.

    My guess is that the JSON is parsed and a huge (given that we're dealing with 5MB) object is created. Node then hits the second debugger, and the inspector needs to load locals. The excruciatingly slow process begins, and the inspector won't show that you've hit a breakpoint until it finishes. So to you it just looks frozen.

    Try replacing your JSON file with something small (like {a:1}). It should load quickly.

    Do you really need to visually inspect the entire object? There are tools better suited for viewing JSON files.