Search code examples
node.jsmongodbnginxcompoundjsjugglingdb

CompoundJS: data returned is incomplete


I'm trying to retrieve a large amount of data (1000+ rows from MongoDB, using the JugglingDB ORM). If I set the limit to be 1001 rows, my data is complete, but once I step up to 1002, the data is incomplete (doesn't matter if I hit it with cURL or the browser). I'm not entirely sure what the issue is, as the console.log shows all of my data, but it sounds like there might be an issue with my response headers or the response itself... here's the code that I'm trying to work with:

function getAllDevices(controller) {
  controller.Device.all({limit: parseInt(controller.req.query.limit)}, function(err, devices) {
    // This shows all of my devices, and the data is correct
    console.log(devices, JSON.stringify(devices).length);
    controller.req.headers['Accept-Encoding'] = '';
    controller.res.setHeader('transfer-encoding', '');
    controller.res.setHeader('Content-Length', JSON.stringify(devices).length);
    return controller.send({success: true, data: devices});
  });
}

Request headers:

Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection  keep-alive
Host    localhost

Response headers:

Access-Control-Allow-Cred...    true
Access-Control-Allow-Head...    X-Requested-With,content-type
Access-Control-Allow-Meth...    GET, POST, OPTIONS, PUT, PATCH, DELETE
Access-Control-Allow-Orig...    *
Connection  keep-alive
Content-Length  1610106
Content-Type    application/json; charset=utf-8
Date    Wed, 20 Aug 2014 17:20:51 GMT
Server  nginx/1.6.0
X-Powered-By    Express

It's good to note that I'm using nginx as a reverse proxy to my Node server, and this is what the config looks like:

location /nodeJs/ {
  proxy_pass http://127.0.0.1:3005/;
  proxy_redirect off;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Connection close;
  proxy_pass_header Content-Type;
  proxy_pass_header Content-Disposition;
  proxy_pass_header Content-Length;
}

Solution

  • I believe I have fixed this issue, and once again, it seems like an nginx change... in my main nginx.conf file, I added:

    gzip on;
    gzip_types application/json;
    

    Adding application/json to the gzip_types is ultimately the solution.