Web Scraping in JavaScript

Question

I am trying to scrape a webpage in JavaScript which looks as follows:

The code shown is part of a larger loop, that loops through each repo and scrapes it's contents. I've confirmed that I'm able to capture the first element of every repo item on the page (so the javascript of "33-js-concepts", the react of "playground", the react of "react-google-static", etc.) and can scrape the all items in the first repo (so javascript, concepts, nodejs, react, angular, etc.) but keep getting this error with subsequent loops. Here is my code:

r.topic = []; // topics used in the repo:
var topics = $('.topics-row-container > a', parent);
    if(topics && topics.length > 0) {
      for (var i in topics) {
        r.topic.push(topics[i].children[0].data.replace(/^\s+|\s+$/g, ''));
        
    }
    console.log(r.topic);

The first loop produces the expected result, with console.log(r.topic) printing:

[
  'javascript',
  'concepts',
  'nodejs',
  'react',
  'angular',
  'programming',
  'javascript-programming'
]

But subsequent loops produce the following error:

r.topic.push(topics[i].children[0].data.replace(/^\s+|\s+$/g, ''));
                                       ^
TypeError: Cannot read property '0' of undefined

I'm new to javascript so am thinking I'm missing something obvious but I can't understand why children would be throwing this error. I even tried making it so children would increment by one with each loop, but I still saw the same error.

I would really appreciate any help!

UPDATE: topics printed to the console looks as follows:

children: [ [Node] ],
    parent: Node {
      type: 'tag',
      name: 'div',
      namespace: 'http://www.w3.org/1999/xhtml',
      attribs: [Object: null prototype],
      'x-attribsNamespace': [Object: null prototype],
      'x-attribsPrefix': [Object: null prototype],
      children: [Array],
      parent: [Node],
      prev: [Node],
      next: [Node]
    },
    prev: Node {
      type: 'text',
      data: '\n          ',
      parent: [Node],
      prev: [Node],
      next: [Circular *7]
    },
    next: Node {
      type: 'text',
      data: '\n      ',
      parent: [Node],
      prev: [Circular *7],
      next: null
    }
  },
  options: { xml: false, decodeEntities: true },
  _root: <ref *8> initialize {
    '0': Node {
      type: 'root',
      name: 'root',
      parent: null,
      prev: null,
      next: null,
      children: [Array],
      'x-mode': 'no-quirks'
    },

But in this particular case, there is no need to parse scrape anything. Github has a public api, even cross origin enabled. Thus you can use fetch and few lines to obtain the result you want. — NVRM
– NVRM, Commented Mar 20, 2021 at 3:26
@fortuneee No, I'm getting a reference error stating: "ReferenceError: document is not defined" and I can't figure out out how to define the document. The parent node is defined like this: var parent = '.source:nth-child(' + i +') '; — Erin
– Erin, Commented Mar 21, 2021 at 14:11
@John this is one such page: github.com/leonardomso?tab=repositories . Any repo page with the "?tab=repositories" extension is what I'm looking at — Erin
– Erin, Commented Mar 29, 2021 at 13:13

Nelson Teixeira · Accepted Answer · 2021-03-25 20:54:34Z

1

+50

If you just need the info now and this isn't part of a larger site that will do this routinely, you could just:

if (topics[i] && topics[i].children && 
    topics[i].children[0] && topics[i].children[0].data)
    r.topic.push(topics[i].children[0].data.replace(/^\s+|\s+$/g, ''));

It's not finding some element. If you want to really look for what it's happening in order to leave this working for all cases, you could:

r.topic = []; // topics used in the repo:
var topics = $('.topics-row-container > a', parent);
try {
    if(topics && topics.length > 0) {
        for (var i in topics) {
            r.topic.push(topics[i].children[0].data.replace(/^\s+|\s+$/g, ''));
        }
        console.log(r.topic);
    }
} catch(error) {
    console.log(error, topics);
}

Then when it fails, you can check the topic structure and see where it failed, so you can enhance you loop to treat that specific case. I could make an working example if you can provide the site you're running this against or the contents of the topics var both when it succeds and when it fails.

If you decide to share this info with us, please don't post it on the question. Use pastebin.com or something.

answered Mar 25, 2021 at 20:54

Nelson Teixeira

6,6366 gold badges46 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Erin Over a year ago

@ Nelson thank you so much! That was exactly it.

fortunee · Accepted Answer · 2021-03-20 03:47:28Z

0

This $('.topics-row-container > a', parent); mostly like doesn't return an array of all those elements, which sorta result in a loop of an object as opposed to an array, when you do for/in.

You need a way to return an array of all these 👉 '.topics-row-container > a' elements.

And you can use document.querySelectorAll(),

So, technically this line:

var topics = $('.topics-row-container > a', parent);

could look something like:

var topics = parent.querySelectorAll('.topics-row-container > a');

answered Mar 20, 2021 at 3:47

fortunee

4,3622 gold badges23 silver badges29 bronze badges

2 Comments

Erin Over a year ago

Right! $('.topics-row-container > a', parent); is not returning an array, that's why I have: topics[i].children[0].data.replace(/^\s+|\s+$/g, ''). <- this does return an array on the first loop, but not subsequent loops. I tried your suggested answer and got "TypeError: parent.querySelectorAll is not a function" :( Thank you so much for your input though! @fortunee

fortunee Over a year ago

Try document.querySelectorAll()

NVRM · Accepted Answer · 2021-03-20 03:53:35Z

0

Base example, fetching the Github api for repositories tagged javascript.

fetch('https://api.github.com/search/repositories?q=javascript')
  .then(v => v.json()).then((v) => {
     console.log(v)
  }
)

answered Mar 20, 2021 at 3:53

NVRM

13.5k2 gold badges102 silver badges102 bronze badges

1 Comment

Erin Over a year ago

I didn't want to use the public api as it has a few limitations I was hoping to bypass. But thank you so much for your input! @NVRM

roschaefer · Accepted Answer · 2021-03-24 16:25:45Z

0

I'm lacking code and a link to the Github page to reproduce the error. Just by looking at the error message it seems that .children is undefined (because the node has no children?).

How about just skipping these nodes?

r.topic = []; // topics used in the repo:
const topics = $('.topics-row-container > a', parent);
if(topics && topics.length > 0) {
  for (const topic of topics) {
    if(!topic.children) {
      // you could `console.log(topic)` here to debug why `.children` is undefined
      continue
    }
    const [firstChild] = topic.children
    r.topic.push(firstChild.data.replace(/^\s+|\s+$/g, ''));
  }
  console.log(r.topic);
}

answered Mar 24, 2021 at 16:25

roschaefer

416 bronze badges

Collectives™ on Stack Overflow

Web Scraping in JavaScript

4 Answers 4

1 Comment

2 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related