Continue fetching data until result is empty array

Question

I'm using the Selly API and trying to fetch all products, however the Selly API paginates products.

So I need to fetch all products and add to an array and then output the result via JSON.

const base64 = require('base-64');
import fetch from 'isomorphic-unfetch';

export default async (req, res) => {
    res.setHeader('Content-Type', 'application/json');
    try {
        request('products')
            .then(data => data.json())
            .then(json => res.status(200).end(JSON.stringify(json)));
    } catch(err) {
        res.end(JSON.stringify({
            "error": err
        }));
    }
}

const getToken = () => {
    return base64.encode(`${process.env.api.email}:${process.env.api.key}`);
}

const request = async (endpoint = '', page) => {
    const options = {
        headers: {
            Authorization: `Basic ${getToken()}`,
            'User-Agent':  `${process.env.api.email} - https://example.com`
        }
    }
    return fetch(`https://selly.io/api/v2/${endpoint}?page=${page}`, options);
}

I'm trying to fetch https://selly.io/api/v2/products?page=${page} until the return result is an empty array [].

My attempt was this:

const repeatedRequest = async (data, endpoint) => {
    let i = 1;
    while (data.length > 0) {
        console.log(data);
        await request(endpoint, i).then((res) => {
            data.push(res);
            i++;
        });
    }
    return data;
}

However this didn't work because the return result was an empty array. I am also confused how I would do this along with the following snippet:

request('products')
    .then(data => data.json())
    .then(json => res.status(200).end(JSON.stringify(json)));

How would I work in the function into this?

How can I do this?

That loop looks bogus. while (data.length >0) /*push something into data*/. That seems like an endless loop, assuming it ever gets started. data will only ever get longer, right? I don't see how data.length might become 0. I was expecting an exit condition where the request detects failure (maybe res is undefined in this case?) and then have that cause the loop to exit (probably just cause it to return when res is undefined.) Or do I misunderstand? — Wyck
– Wyck, Commented Sep 5, 2019 at 3:03
Wyck I think I'm confused as you are. How would you do this? — kinx
– kinx, Commented Sep 5, 2019 at 3:10
The problem, as Wyck says, is that data.length will always be greater than 0, assuming it starts nonempty (and if it starts out empty, the loop will never run at all). It seems you want to check if the actual API response res has length 0, instead of data, which is where you are accumulating all of the responses. — jacob
– jacob, Commented Sep 5, 2019 at 3:14
I think you can figure this out - maybe try adding some more console logging to debug will help. Currently there's a console.log inside the while loop - when you run it, does it print anything out? — jacob
– jacob, Commented Sep 5, 2019 at 3:30

Wyck · Accepted Answer · 2020-07-10 04:35:08Z

You asked (in a comment) a sub-question about how to set up the loop to call the request function over and over again to get results until there are no more results. I'll try to answer your sub-question here in a minimal and general-purpose way that will hopefully be of use to you and others.

Here's one way of setting up the asynchronous loop mechanism in isolation. There's always more than one way. ;)

I will craft a couple of utility functions that allow me to be express the solution concisely.

I need a fake request function that will return some number of results, then return something falsey. I'll call this fakeRequest.
I need a general purpose way to call an async function with an ever-increasing integer index argument until it returns something falsey as an async generator function. I'll call this iterateAllAsync
I need a general purpose way to coalesce all the results of an async generator function into an array. Essentially an async version of Array.from. I'll call this arrayFromAsync

Then you can say: Make fake requests starting from i = 0 and incrementing i until it doesn't return anything and then return all those results as an array.

...and you can express this (roughly) as: arrayFromAsync(iterateAllAsync(fakeRequest)). In practice I'll probably add a lambda to show you how to curry some extra parameters into the fakeRequest, but this is the basic structure of it.

Here's the whole thing:

// An example fake request function
const fakeRequest = async (endpoint, i) => {
  const kNumberOfFakeResultsToReturn = 3;
  if (i >= kNumberOfFakeResultsToReturn) return;
  return { content: `fake payload #${i} from ${endpoint}` };
};

// A general purpose utility function:
// yields results of calling fn(i) with ever increasing i until it returns something falsey
async function* iterateAllAsync(fn, i = 0) {
  while (true) {
    let res = await fn(i)
    if (!res) return;
    yield res;
    ++i;
  }
}

// A general purpose utility function:
// returns an async generator's results as an array
const arrayFromAsync = async (it) => {
  let results = [];
  for await (let res of it) {
    results.push(res);
  }
  return results;
}

// Here's where we describe what we want done:
arrayFromAsync(iterateAllAsync((i) => fakeRequest('fakeEndpoint', i)))
  .then(console.log);

This code gives me an error in arrayFromAsync because res is not defined. Should it be for await (let res of it)?

Collectives™ on Stack Overflow

Continue fetching data until result is empty array

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related