Patterns for async list processing in node.js

Question

I have an application where a database query returns a number of rows (typically, less than 100). For each row, I need to make an http call to get supplemental data. I'd like to fire off all of the requests, and then when the last callback completes, move on to rendering the result page.

So far, the answers to similar questions I've looked at have either been chaining the requests by making request #2 in the callback for request #1 (advantages: simple, avoids burying the server in multiple requests), or by firing all of the requests with no tracking of whether all of the requests have completed (works well in the browser where the callback updates the UI).

My current plan is to keep a counter of requests made and have the callback decrement the counter; if it reaches zero, I can call the render function. I may also need to handle the case where responses come in faster than requests are being made (not likely, but a possible edge case).

Are there are other useful patterns for this type of problem?

queue function in async module is exactly what you're looking for. — Dogbert
– Dogbert, Commented Apr 15, 2013 at 13:39

TheHippo · Accepted Answer · 2013-04-15 14:13:06Z

1

When using async code could roughly look like this:

var async = require('async');

results = [];

var queue = async.queue(function(row, callback) {
    http.fetchResultForRow(row, function(data){
        result.push(data);
        callback();      
    });
}, 1);

queue.drain = function() {
    console.log("All results loaded");
    renderEverything(results);
}

database.fetch(function(rows) {
    for (var i=0; i < rows.length; i++) {
        queue.push(rows[i]);
    }
});

If the order does not matter you also could use: map

Look around in the documenation of async, there are a lot of useful patterns.

answered Apr 15, 2013 at 14:13

TheHippo

63.3k15 gold badges77 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Michael Anderson · Accepted Answer · 2013-04-15 14:20:44Z

0

You can implement this quite nicely with promises using the when library. Though if you want to rate limit the calls to getting the extra info you will need to do a little more work than in the async approach of TheHippo I think.

Here's an example:

when = require('when')

// This is the function that gets the extra info.
// I've added a setTimeout to show how it is async.    
function get_extra_info_for_row(x, callback) {
   setTimeout( function(){ return callback(null, x+10); }, 1 );
};

rows = [1,2,3,4,5];

row_promises = rows.map(
   function(x) {
      var defered = when.defer()
      get_extra_info_for_row(x, function(err,extra_info) {
         if(err) return defered.reject(err);
         defered.resolve([x,extra_info]);
      });
      return defered.promise;
   })


when.all( row_promises )
   .then(
      function(augmented_rows) { console.log( augmented_rows ); },
      function(err) { console.log("Error", err ); }
      );

This outputs

[ [ 1, 11 ], [ 2, 12 ], [ 3, 13 ], [ 4, 14 ], [ 5, 15 ] ]

answered Apr 15, 2013 at 14:20

Michael Anderson

74.3k9 gold badges147 silver badges196 bronze badges

1 Comment

TomG Over a year ago

I decided to go with async since it has builtin rate limiting, but this looks 'promising', too.

Collectives™ on Stack Overflow

Patterns for async list processing in node.js

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related