The best pattern for handling async looping in Node.js

Question

I'm new to Node and trying to ensure that I'm using sane designs for a JSON-driven web app.

I've got a bunch of data stored in Redis and am retrieving it through node, streaming out the results as they come from Redis. Here's a good example of what I'm doing:

app.get("/facility", function(req, res) {
    rc.keys("FACILITY*", function(err, replies) {
        res.write("[");
        replies.forEach(function (reply, i) {
            rc.get(reply, function(err, reply) {
                res.write(reply);
                if (i == replies.length-1) {
                    res.write("]");
                    res.end();
                }
                else
                    res.write(",");
            });
        });
    });
});

Essentially I'm getting set of keys from Redis and then requesting each one, streaming out the result into semi-manually created JSON (the strings coming out of Redis are already in JSON). Now this works nicely, but I can't help thinking that the i == replies.length-1 is a little untidy?

I could do all this with mget in Redis, but that isn't really the point I'm trying to get it; it's how best to handle async looping with forEach, streaming the output and gracefully closing off the connection with res.end with the looping is done.

Is this the best way, or is there a more elegant pattern I could follow?

for deep nested function callbacks i would use the async.js library. — BRampersad
– BRampersad, Commented Dec 20, 2011 at 17:45

s4y · Accepted Answer · 2011-12-21 17:13:07Z

6

The above code might not do what you expect. You're kicking off each .get() in sequence, but they might not call back in sequence — so the results could stream out in any order. If you want to stream the results instead of collecting them in memory, you need to .get() in sequence.

I think that caolan’s async library makes a lot of this easier. Here’s one way you could use it to get each item in sequence (warning, untested):

app.get("/facility", function(req, res) {
    rc.keys("FACILITY*", function(err, replies) {
        var i = 0;
        res.write("[");
        async.forEachSeries(replies, function(reply, callback){
            rc.get(reply, function(err, reply) {
                if (err){
                    callback(err);
                    return;
                }
                res.write(reply);
                if (i < replies.length) {
                    res.write(",");
                }
                i++;
                callback();
            });
        }, function(err){
            if (err) {
                // Handle an error
            } else {
                res.end(']');
            }
        });
    });
});

If you don’t care about the order, just use async.forEach() instead.

If you wouldn’t mind collecting the results and want them to return in sequence, you could use async.map() like this (warning, also untested):

app.get("/facility", function(req, res) {
    rc.keys("FACILITY*", function(err, replies) {
        async.map(replies, rc.get.bind(rc), function(err, replies){
            if (err) {
                // Handle an error
            } else {
                res.end('[' + replies.join(',') + ']');
            }
        });
    });
});

edited Dec 21, 2011 at 17:13

answered Dec 20, 2011 at 18:25

s4y

51.9k12 gold badges73 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Matt S. Over a year ago

That's great; thanks for the code; I tried out the async library and it works perfectly. Order doesn't matter, but the map solution looks far more elegant, so I may just stick to that.

Matt S. Over a year ago

I'm trying to work out how the rc.get.bind(rc) parameter in the map function call works; it a nifty way of doing it. Could you explain a bit about how that works exactly?

s4y Over a year ago

@mjs bind is part of ECMAScript 5, and it returns a copy of a function which is “bound” to a particular this value. In this case, it means that when async.map calls the get(), it will have rc as its this value.

s4y Over a year ago

@mjs Normally, the value of this inside a function is determined by how you call it — calling a function on an object (like rc.get()) sets the value of this inside the function to rc. That’s determined at call time. If you were to do something like var cb = rc.get; cb() (which is the effect of passing it as an argument to async.map), it runs without the context of rc. Likewise, if you were to do var foo = {}; foo.get = rc.get; foo.get();, get would have foo as its context, not rc, and would probably behave very, very badly.

Matt S. Over a year ago

Thanks for the expanded explanation; appreciate it.

|

alessioalex · Accepted Answer · 2011-12-20 17:48:15Z

You can use the async library, it provides some handy methods for looping, such as forEach:

forEach(arr, iterator, callback)

Applies an iterator function to each item in an array, in parallel. The iterator is called with an item from the list and a callback for when it has finished. If the iterator passes an error to this callback, the main callback for the forEach function is immediately called with the error.

Note, that since this function applies the iterator to each item in parallel there is no guarantee that the iterator functions will complete in order.

Example

// assuming openFiles is an array of file names and saveFile is a function
// to save the modified contents of that file:

async.forEach(openFiles, saveFile, function(err){
    // if any of the saves produced an error, err would equal that error
});

chjj · Accepted Answer · 2011-12-20 18:09:39Z

but I can't help thinking that the i == replies.length-1 is a little untidy?

I've heard a lot of people say that. This is how I would do it by hand:

app.get("/facility", function(req, res, next) {
  rc.keys("FACILITY*", function(err, replies) {
    if (err) return next(err);
    var pending = replies.length;
    res.write("[");
    replies.forEach(function (reply) {
      rc.get(reply, function(err, reply) {
        res.write(reply);
        if (!--pending) {
          res.write("]");
          return res.end();
        }
        res.write(",");
      });
    });
  });
});

Obviously doing it by hand isn't too pretty, which is why people have it abstracted into a library or some other function. But like it or not, that is how you do an async parallel loop. :)

You can use the async library mentioned before to hide the nasty innards.

Collectives™ on Stack Overflow

The best pattern for handling async looping in Node.js

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related