Search code examples
javascriptnode.js

Unable to add new properties in nested callback while joining MongoDB collections in Node.js


I'm trying to implement a JOIN-like operation between two MongoDB collections in Node.js/Express: persons and addresses. Each person has an id_adr field referencing their address. Here's my current code:

var ret;
app.get('/tp/show/method', function (req, res) {
    ret = {};
    User2Model.find(function(err, users) {
        if(err) {
            console.error('Error finding users: ' + err);
        }
        var last_id;
        for(var user_id in users) {
            last_id = users[user_id]._id;
            ret[last_id] = users[user_id];
        }
        
        for(var user_id in users) {
            AdrModel.find({ 'user_id': users[user_id]['id_adr'] }, function(err, adr) {
                if (err) {
                    console.error('Error finding addresses: ' + err);
                }
                for(var i in adr) {
                    for(var user_id in users) {
                        if (users[user_id].id_adr == adr[i].user_id) {
                            // Trying to add address information to the user object
                            ret[users[user_id]._id] = adr;
                            
                            if(users[user_id]._id == last_id) {
                                var url_parts = url.parse(req.url, true);
                                var query = url_parts.query;
                                res.setHeader('content-type', 'application/javascript');
                                json = query.callback + '(' + JSON.stringify({
                                    data: {success: 1, value: ret}
                                }) + ')';
                                res.send(json);
                            }
                            break;
                        }
                    }
                }
            });
        }
    });
});

Problem: Even though ret is defined in the outer scope, I'm unable to add new properties to the objects within it from the nested callback. When I try to add address information, it only works when overwriting existing properties, but not when trying to add new ones like "addr".

Expected Behavior: I want to be able to add the address information as a new property to each user object in the ret variable.

Question: What am I doing wrong, and how can I properly add new properties to the objects within ret from inside the nested callback?

Additional Context:

  • Using Node.js with Express
  • MongoDB with Mongoose
  • Implementing JSONP response

Solution

  • As I indicated in comments to @Tomalak's solution above, events can also be used to manage program flow in an async environment.

    Here's an alternate partial solution that uses just that approach.

    Please note that of the various ways of accomplishing this goal (I know of at least three, or four if you accept that the pain of "Callback Hell" can be ameliorated through the use of callbacks defined outside of and only referenced inline by their caller), I prefer using events since they are a more natural way for me to think about this class of problem.

    Take-aways

    1. Events are an efficient and easily understandable way to manage program flow in an async programming environment.
    2. Rather than simple triggers, events can be used transport any data so they can be used further on for any purpose.
    3. Events can easily call other events without worrying about scope.
    4. Event processing allows you to unwind your code such that it becomes easier to track, and thus debug, as well as reducing the burden on the stack typically seen in deeply nested or recursive code. In other words, events are fast and very memory efficient.

    Explanation

    The code first defines two mocks:

    1. an App class which provides a get method, allowing us to mock out the OP's app instance, and
    2. a User2Model singleton that provides a find function for the same purpose.

    It then documents the following events:

    1. error - which is called on any errors to print a message to console and exit the program
    2. get - which is fired with the result of the app.get method and immediately fires the processUsers event with {req:req,res:res}
    3. processUsers - fired by the get event handler with a mocked array of user objects, sets up a results object and a last_id value, and then calls the nextUser event.
    4. nextUser - fired by the processUsers event which picks the next user off the users array, sets evt.last_id, adds the user to the evt.results, and emits itself, or if there are no users left on the evt.users array, emits complete
    5. complete - fired by nextUser and simply prints a message to console.

    Event handlers are next defined using the 'on'+eventName convention.

    And finally, we

    1. define an eventHandlers object, to map handlers to their appropriate events,
    2. instantiate our app instance, and
    3. invoke its get method with a callback that simply emits a get event to start the ball rolling.

    I've documented most of the solution using jsdoc and added logging messages to show progress as each event is emitted and its handler invoked. The result of the run is included after the code. (The http req and res objects have been commented out of the log messages for the sake of brevity.)

    One final note, while this example is 269 lines long, most of it is documentation.

    The actual code (without the mocks) is only about 20 or 25 lines.

    Code

    /*
    
     Example of using events to orchestrate program flow in an async
     environment.
    
     */
    
    var util = require('util'),
        EventEmitter = require('events').EventEmitter;
    
    // mocks
    
    /**
     * Class for app object (MOCK)
     * @constructor
     * @augments EventEmitter
     */
    var App = function (handlers) {
      EventEmitter.call(this);
      this.init(handlers);
    };
    util.inherits(App, EventEmitter);
    
    /**
     * Inits instance by setting event handlers
     *
     * @param {object} handlers
     * @returns {App}
     */
    App.prototype.init = function (handlers) {
      var self = this;
      // set event handlers
      Object.keys(handlers).forEach(function (name) {
        self.on(name, handlers[name]);
      });
      return self;
    };
    
    /**
     * Invokes callback with req and res
     * @param uri
     * @param {App~getCallback} cb
     */
    App.prototype.get = function (uri, cb) {
    
      console.log('in app.get');
    
      var req = {uri: uri},
          res = {uri: uri};
      /**
       * @callback App~getCallback
       * @param {object} req - http request
       * @param {object} res - http response
       * @fires {App#event:get}
       */
      cb(req, res);
    };
    
    /**
     * Data access adapter - (MOCK)
     * @type {object}
     */
    var User2Model = {};
    /**
     *
     * @param {User2Model~findCallback} cb
     */
    User2Model.find = function (cb) {
      var err = null,
          users = [
            {_id: 1},
            {_id: 2}
          ];
      /**
       * @callback User2Model~findCallback
       * @param {Error} err
       * @param {Array} users
       */
      cb(err, users);
    };
    
    
    // events
    
    /**
     * Error event.
     *
     * @event App#error
     * @type {object}
     * @property {object} [req] - http request
     * @property {object} [res] - http response
     * @property {string} where - name of the function in which the error occurred
     * @property {Error} err - the error object
     */
    
    /**
     * Get event - called with the result of app.get
     *
     * @event App#get
     * @type {object}
     * @property {object} req - http request
     * @property {object} res - http response
     */
    
    /**
     * ProcessUsers event - called
     *
     * @event App#processUsers
     * @type {object}
     * @property {object} req - http request
     * @property {object} res - http response
     * @property {Array} users - users
     */
    
    /**
     * NextUser event.
     *
     * @event App#nextUser
     * @type {object}
     * @property {object} req - http request
     * @property {object} res - http response
     * @property {Array} users
     * @property {*} last_id
     * @property {object} result
     */
    
    /**
     * Complete event.
     *
     * @event App#complete
     * @type {object}
     * @property {object} req - http request
     * @property {object} res - http response
     * @property {Array} users
     * @property {*} last_id
     * @property {object} result
     */
    
    // event handlers
    
    /**
     * Generic error handler
     *
     * @param {App#event:error} evt
     *
     * @listens App#error
     */
    var onError = function (evt) {
      console.error('program error in %s: %s', evt.where, evt.err);
      process.exit(-1);
    };
    
    /**
     * Event handler called with result of app.get
     *
     * @param {App#event:get} evt - the event object
     *
     * @listens App#appGet
     * @fires App#error
     * @fires App#processUsers
     */
    var onGet = function (evt) {
      console.log('in onGet');
      var self = this;
      User2Model.find(function (err, users) {
        if (err) {
          console.log('\tonGet emits an error');
          return self.emit('error', {
            res:evt.res,
            req:evt.req,
            where: 'User2Model.find',
            err: err
          });
        }
        self.emit('processUsers', {
          //req:req,
          //res:res,
          users: users
        });
      });
    };
    
    /**
     * Handler called to process users array returned from User2Model.find
     *
     * @param {App#event:processUsers} evt - event object
     * @property {object} req - http request
     * @property {object} res - http response
     * @property {Array} users - array of Users
     *
     * @listens {App#event:processUsers}
     * @fires {App#event:nextUser}
     */
    var onProcessUsers = function (evt) {
      console.log('in onProcessUsers: %s', util.inspect(evt));
      var self = this;
      evt.last_id = null;
      evt.result = {};
      self.emit('nextUser', evt);
    };
    
    /**
     * Handler called to process a single user
     *
     * @param evt
     * @property {Array} users
     * @property {*} last_id
     * @property {object} result
     *
     * @listens {App#event:nextUser}
     * @emits {App#event:nextUser}
     * @emits {App#event:complete}
     */
    var onNextUser = function (evt) {
      var self = this;
    
      console.log('in onNextUser: %s', util.inspect(evt));
    
      if (!(Array.isArray(evt.users) && evt.users.length > 0)) {
        return self.emit('complete', evt);
      }
    
      var user = evt.users.shift();
    
      evt.last_id = user._id;
    
      evt.result[evt.last_id] = user;
    
      self.emit('nextUser', evt);
    };
    
    /**
     * Handler invoked when processing is complete.
     *
     * @param evt
     * @property {Array} users
     * @property {*} last_id
     * @property {object} result
     */
    var onComplete = function (evt) {
      console.log('in onComplete: %s', util.inspect(evt));
    };
    
    // main entry point
    
    var eventHandlers = { // map our handlers to events
      error: onError,
      get: onGet,
      processUsers: onProcessUsers,
      nextUser: onNextUser,
      complete: onComplete
    };
    
    var app = new App(eventHandlers); // create our test runner.
    
    app.get('/tp/show/method', function (req, res) { // and invoke it.
      app.emit('get', {
        req: req,
        res: res
      });
      /* note:
           For this example, req and res are added to the evt
           but are ignored.
    
           In a working application, they would be used to
           return a result or an error, should the need arise,
           via res.send().
       */
    });
    

    Result

    in app.get
    in onGet
    in onProcessUsers: { users: [ { _id: 1 }, { _id: 2 } ] }
    in onNextUser: { users: [ { _id: 1 }, { _id: 2 } ], last_id: null, result: {} }
    in onNextUser: { users: [ { _id: 2 } ],
        last_id: 1,
        result: { '1': { _id: 1 } } }
    in onNextUser: { users: [],
        last_id: 2,
        result: { '1': { _id: 1 }, '2': { _id: 2 } } }
    in onComplete: { users: [],
        last_id: 2,
        result: { '1': { _id: 1 }, '2': { _id: 2 } } }