MongoDB and Node js Asynchronous programming

Question

I am trying to solve an exam problem, so I cannot post my exam code as it is. So I have simplified such that it addresses the core concept that I do not understand. Basically, I do not know how to slow down node's asynchronous execution so that my mongo code can catch up with it. Here is the code:

MongoClient.connect('mongodb://localhost:27017/somedb', function(err, db) {
  if (err) throw err;
  var orphans  = [];
  for (var i; i < 100000; i++) {
    var query = { 'images' : i };
    db.collection('albums').findOne(query, function(err, doc_album) {
        if(err) throw err;
        if (doc_album === null) {
          orphans.push(i);
        }
    });
  }
  console.dir(orphans.length);
  return db.close();
});

So I am trying to create an array of those images who do not match my query criteria. I end up with a orphans.length value of 0 since Node does not wait for the callbacks to finish. How can I modify the code such that the callbacks finish executing before I count the number of images in the array that did not meet my query criteria?

Thanks in advance for your time.

Bharat

'images': {$in: [...]}

Prinzhorn
– Prinzhorn

2013-09-27 07:34:35 +00:00
Commented Sep 27, 2013 at 7:34 — Prinzhorn
– Prinzhorn, Commented Sep 27, 2013 at 7:34

Andrey Sidorov · Accepted Answer · 2013-09-27 22:38:12Z

2

I assume you want to do 100000 parallel DB calls. To "wait" 10000 calls completion in each call callback we increase finished calls counter and invoke main callback when last one finished. Note that very common mistake here is to use for loop variable as a closure inside callback. This does not work as expected as all 10000 handlers scheduled first and by the time first is executed loop variable is of the same, maximum value.

function getOrphans(cb) {
  MongoClient.connect('mongodb://localhost:27017/somedb', function(err, db) {
    if (err) cb(err);
    var orphans  = [];

    var numResponses = 0;
    var maxIndex = 100000
    for (var i = 0; i < maxIndex; i++) {
       // problem: by the time you get reply "i" would be 100000.
       // closure variable changed to function argument:  
       (function(index) {
        var query = { 'images' : index };
        db.collection('albums').findOne(query, function(err, doc_album) {
          numResponses++;
          if(err) cb(err);
          if (doc_album === null) {
            orphans.push(index);
          }
          if (numResponses == maxIndex) {
            db.close();
            cb(null, orphans);
          }
        });
      })(i); // this is "immediately executed function
    }
  });
 }


 getOrphans(function(err, o) {
   if (err)
     return console.log('error:', err);
   console.log(o.length);
 });

edited Sep 27, 2013 at 22:38

answered Sep 27, 2013 at 7:26

Andrey Sidorov

25.6k5 gold badges65 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Bharat Over a year ago

This works. Thanks. Question: so that reason that this is working is because of the "immediately executed function" as you show in the comment section above? Could you please elaborate it further.

Adowrath Over a year ago

Could this be solved easier/more intuitively with the new let, as in for (let i = 0; i < maxIndex; i++) { ... }?

6502 · Accepted Answer · 2013-09-27 05:40:29Z

Im not suggesting this is the best way to handle this specific problem in Mongo, but if you need to wait to the DB to reply before continuing then just use the callback to start next request.

This is not obvious at first, but you can refer to the result processing function inside the function itself:

var i = 0;
var mycback = function(err, doc_album) {
    // ... process i-th result ...
    if (++i < 100000) {
        db.collections("album").findOne({'images': i}, mycback);
    } else {
        // request is complete, "return" result
        result_cback(null, res);
    }
};
db.collections('album').findOne({'images': 0}, mycback);

This also means that your function itself will be async (i.e. will want a result_cback parameter to call with the result instead of using return).

Writing a sync function that calls an async one is just not possible.

You cannot "wait" for an event in Javascript... you must set up an handler for the result and then terminate.

Waiting for an event is done in event-based processing by writing a "nested event loop" and this is for example how message boxes are handled in most GUI frameworks. This is a capability that Javascript designers didn't want to give to programmers (not really sure why, though).

painotpi · Accepted Answer · 2013-09-27 05:18:10Z

0

Since you know it does not wait for the call to come back. You can do the console.dir inside your callback function, this should work (although I haven't tested it)

    db.collection('albums').findOne(query, function(err, doc_album) {
        if(err) throw err;
        if (doc_album === null) {
          orphans.push(i);
        }
        console.dir(orphans.length);
    });

answered Sep 27, 2013 at 5:18

painotpi

7,0122 gold badges40 silver badges72 bronze badges

Comments

Josh C. · Accepted Answer · 2013-09-27 05:18:53Z

0

You don't need to slow anything down. If you are simply trying to load 100,000 images from the albums collection, you could consider using the async framework. This will let you assign tasks until the job is complete.

Also, you probably don't want request 100,000 records one-by-one. Instead, you probably want to page them.

answered Sep 27, 2013 at 5:18

Josh C.

4,3935 gold badges33 silver badges52 bronze badges

Collectives™ on Stack Overflow

MongoDB and Node js Asynchronous programming

4 Answers 4

2 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related