How to run multiple asynchronous functions then execute callbacks?

Question

In my Node.js code I need to make 2 or 3 API calls, and each will return some data. After all API calls are complete, I want to collect all the data into a single JSON object to send to the frontend.

I know how to do this using the API callbacks (the next call will happen in the previous call's callback) but this would be slow:

//1st request
request('http://www.example.com', function (err1, res1, body) {
  
  //2nd request
  request('http://www.example2.com', function (err2, res2, body2) {
  
    //combine data and do something with it

  });

});

I know you could also do something similar and neater with promises, but I think the same concept applies where the next call won't execute until the current one has finished.

Is there a way to call all functions at the same time, but for my final block of code to wait for all API calls to complete and supply data before executing?

This is a limitation of callback based asynchronous functions. There just isn't a super clean way to do it. It's trivial with promises though. — Madara's Ghost
– Madara's Ghost, Commented Sep 28, 2015 at 17:19
Not a duplicate, but closely related: How can I fetch an array of URLs with Promise.all?. — Henke
– Henke, Commented May 20, 2021 at 11:03

Madara's Ghost · Accepted Answer · 2018-10-04 11:58:54Z

51

Promises give you Promise.all() (this is true for native promises as well as library ones like bluebird's).

Update: Since Node 8, you can use util.promisify() like you would with Bluebird's .promisify()

var requestAsync = util.promisify(request); // const util = require('util')
var urls = ['url1', 'url2'];
Promise.all(urls.map(requestAsync)).then(allData => {
    // All data available here in the order of the elements in the array
});

~~So what you can do (native):~~

function requestAsync(url) { return new Promise(function(resolve, reject) { request(url, function(err, res, body) { if (err) { return reject(err); } return resolve([res, body]); }); }); } Promise.all([requestAsync('url1'), requestAsync('url2')]) .then(function(allData) { // All data available here in the order it was called. });

If you have bluebird, this is even simpler:

var requestAsync = Promise.promisify(request);
var urls = ['url1', 'url2'];
Promise.all(urls.map(requestAsync)).then(allData => {
    // All data available here in the order of the elements in the array
});

edited Oct 4, 2018 at 11:58

answered Sep 28, 2015 at 17:23

Madara's Ghost

176k52 gold badges274 silver badges314 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

thefourtheye Over a year ago

I guess we cannot avoid Promise Constructor Anti-pattern here :'(

jfriend00 Over a year ago

@thefourtheye - it's not an anti-pattern when you're wrapping a thing that doesn't return a promise. Somebody has to make the promise. No other option unless you have a library function that does the promisifying wrapper for you.

Madara's Ghost Over a year ago

@thefourtheye You can easily make a promisify function similar to bluebird's, to hide away the ugliness.

Ben · Accepted Answer · 2015-09-28 23:03:45Z

14

Sounds like async.parallel() would also do the job if you'd like to use async:

var async = require('async');

async.parallel({
    one: function(parallelCb) {
        request('http://www.example1.com', function (err, res, body) {
            parallelCb(null, {err: err, res: res, body: body});
        });
    },
    two: function(parallelCb) {
        request('http://www.example2.com', function (err, res, body) {
            parallelCb(null, {err: err, res: res, body: body});
        });
    },
    three: function(parallelCb) {
        request('http://www.example3.com', function (err, res, body) {
            parallelCb(null, {err: err, res: res, body: body});
        });
    }
}, function(err, results) {
    // results will have the results of all 3
    console.log(results.one);
    console.log(results.two);
    console.log(results.three);
});

answered Sep 28, 2015 at 23:03

Ben

5,0842 gold badges21 silver badges23 bronze badges

3 Comments

Rajeev Jayaswal Over a year ago

And just one question here, can we not write those parallel function one, two and three inside for loop ?

Ben Over a year ago

I'm not sure what you meant by inside the loop. My guess is this whole async.parallel() inside a loop? I believe It should be ok if async.each() is used instead of using native loops.

Rajeev Jayaswal Over a year ago

I figured it out what I needed. I was talking about async.map(handleList, findStatus, function(err, results) { ... }); where findStatus is function that will be called as many times as many items are present in handleList.

Kaushik Thanki · Accepted Answer · 2020-10-16 15:00:38Z

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

Promise.all is now included with ES6 so you don't need any 3rd party libraries at all.

"Promise.all waits for all fulfillments (or the first rejection)"

I've setup a gist to demonstrate Promise.all() with refactoring itterations at: https://gist.github.com/rainabba/21bf3b741c6f9857d741b69ba8ad78b1

I'm using an IIFE (Immediately Invoked Function Expression). If you're not familiar, you'll want to be for the example below though the gist shows how with using an IIFE. https://en.wikipedia.org/wiki/Immediately-invoked_function_expression

TL;DR

( function( promises ){
    return new Promise( ( resolve, reject ) => {
        Promise.all( promises )
            .then( values => {
                console.log("resolved all promises")
                console.dir( values );
                resolve( values.reduce( (sum,value) => { return sum+value }) ); //Use Array.prototype.reduce() to sum the values in the array
            })
            .catch( err => {
                console.dir( err );
                throw err;
            });

    });
})([ 
    new Promise( ( resolve, reject ) => {
        console.log("resolving 1");
        resolve( 1 );
    }),
    new Promise( ( resolve, reject ) => {
        console.log("resolving 2");
        resolve( 2 );
    })
 ]).then( sum => { console.dir( { sum: sum } ) } )

Ramgopal Bhat · Accepted Answer · 2021-05-19 12:28:55Z

1

I had a similar use case where I had to do 10 concurrent calls. I did it with the combination of async/await and Promise.all.

async function getData() {
  try {
    let result = null
    const ids = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]

    let promises = ids.map(async (id) => {
        return fetch(
          `https://jsonplaceholder.typicode.com/todos/${id}`
        ).then((data) => data.json());
      });
    
    result = await Promise.all(promises)
    return result
  } catch(err) {
    console.log("error: ", err)
  }
}

getData().then(data => console.log(data))

edited May 19, 2021 at 12:28

answered May 19, 2021 at 12:23

Ramgopal Bhat

114 bronze badges

1 Comment

Henke Over a year ago

Note that you don't need async/await in the function getData(). Remove them both and the code still works just fine. (Try it!) The reason is that getData() returns a Promise which gets resolved by the .then() call. ~ * ~ (A side-note: you don't need any line break at all after , but that's a matter of taste, I would say. :-)

Collectives™ on Stack Overflow

How to run multiple asynchronous functions then execute callbacks?

4 Answers 4

3 Comments

3 Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related