Is there a way I can get links from pages other than the DOM in a firefox extension?

Question

I'm trying to get links from other pages but since they aren't loaded I can't do something like this.

var urlsFound = [];
var url = "https://www.youtube.com";

for(var i = url.links.length; i --> 0;)
{
    urls.push(url.links[i].href);
}

urlsFound.forEach((item, index) =>
{
    console.log(`${index} : ${item}`);
});

Now of course the code above won't work but it's there just to better illustrate what I'm trying to accomplish. How can I get links from other pages without loading them into the DOM while having it work in a firefox extension? I'm willing to use any technology provided it can work within a firefox extension.

Answer 1

You need to make a http request using something like fetch or axios and then parse the html and get the links, using cheerio. Here's an example.

You can see how to use axios and cheerio here:

https://www.npmjs.com/package/axios

https://cheerio.js.org/

const axios = require("axios").default; // can use fetch etc.
const cheerio = require("cheerio");

function getLinks(url) {
  return new Promise((resolve, reject) => {
    axios.get(url).then(res => res.data).then(HTML => {
      const $ = cheerio.load(HTML);
      const links = $("a").toArray();
      resolve(links);
    }).catch(err => reject(err));
  });
}