puppeteer how to click dynamic "View More" link until all content loaded

Question

I’m trying to scrape a page that has a “View More” link on it, which dynamically loads more content. The View More link will be present again if there is even more content to be loaded, until there is no more.

I want to programatically keep clicking the “View More” links in a loop until ALL of the content is loaded, then I'll scrape the full list.

My code below finds the FIRST View More link ok but can’t locate the SECOND (or more) link after if it is dynamically loaded.

Can anyone help?

Thanks.

Loop function to keep clicking the “View More Links” link and load dynamic content

const clickViewMoreLoop = async() => {
    try {
        await page.waitForSelector(viewMoreSelector, { timeout: 5000 });
        console.log(“still loading…”);
        await page.$eval(selector, (el) => {
            el.click();
        });
        await page.waitForNavigation({ waitUntil: 'networkidle0' });
        return clickViewMoreLoop();
    } catch (e) {
        console.log(“loaded");
    }
}

await clickViewMoreLoop();

Answer 1

Ok, so I identified the problem being that the "View More" button needed to be within the current browser viewport before it could be selected and clicked. So I added the el.scrollIntoView(); code to bring it into view before the click event, then a small pause after so the code had time to catch it.

This seems to work ok...

console.log("click view more to load entire list");
while (true) {
    try {
        await page.$eval('#mainContentContainer > main > div > div > div > div.notranslate > div > div:nth-child(2) > button', (el) => {
                el.scrollIntoView();
                el.click();
        });
        await page.waitForTimeout(500);
        console.log("loading...");
    } catch (e) {
        console.log("done");
        break;
    }
}

If anyone as any other improvements please share!