Search code examples
javascriptpuppeteer

Looping inside a page.evaluate in Puppeteer


I have a loop inside a page.evaluate method. The loop iterates a query selector which catches an innerText from multiple instances of a text element in a page.

I am receiving an error Evaluation Failed: Cannot read property of 'innerText'

I tried to loop outside of page.evaluate, but my iteration variable is not accessible from within the page.evaluate function.

// Here's a rough draft of what i'm trying to achieve:

    const scrapeData = [];
    const data = await page.evaluate(() => {

    // Iteration to capture each target text in the page  
    for (var i = 1; i < 9; i++) {

        // Select target text 
        const serpDesc = document
        .querySelector(
          `#rso > div:nth-child(4) > div > div:nth-child(${i}) > div > div > div.s > div > span`
        )
        .innerText.trim();

      // Build an array for the captured text
      scrapeData[i] = serpDesc

      return {
        serpDesc
      };
    };
  });

My goal is to scrape some link descriptions(plain text) from a page into an array. Without the iteration code, everything works fine.


Solution

  • Try:

    const serpDesc = await page.evaluate(
      () => [...document.querySelectorAll(`#rso > div:nth-child(4) > div > div:nth-child(${i}) > div > div > div.s > div > span`)].map(elem => elem.innerText)
    );
    

    You will probably need to reconstruct your selector a bit, or maybe wrap serpDesc function in a for of or forEach loop.

    You could also try something like

    async function elSelector(i) {
                //Where i is the incremented value you pass
                
                await page.evaluate((i) => {
                
                    let eval = $('yourSelector').toArray();
                    $(eval[i]).innerText
                }, i)
    
            }
            
    for (i=0; i<9; i++) {
      elSelector(i);
      }