Scrape a website from chrome console with javascript

Question

I'm trying to scrape a webpage that contains a table of 1000 elements. The webpage updates a tag whenever an element in the table is clicked. From the chrome console, I want to click on each element, wait for the tag to be updated and download the tag. Currently I am doing the following:

for(i = 0; i < 1000; i++){
  document.querySelectorAll('element_in table')[i].click()
  text = document.querySelector('tag_to_read_from').innerHTML
  // download text
}

The problem is that there is a delay in the tag being updated after an element in the table is clicked. As a result, due to the asynchronous nature of javascript, the script is downloading 1000 empty files.

Is there a way to wait for X seconds after clicking on an element, and then download the updated tag?

Answer 1

function scrape (index, max) {
  document.querySelectorAll('element_in table')[index].click();
  
  setTimeout(() => {
    // download text

    if (index < max) scrape(++index, max);
  }, 5000);
}

scrape(0, 1000);

You can use a timeout to delay the logic. Just pick a big enough time.