Cannot click on cookie button with puppeteer

I am trying to close a cookie banner /modal on a site and it seems to me clicking on button and a tag elements with puppeteer doesn't work for me. The cookie banner/modal always stays in the screenshot I take. Also when I log the button or use an if condition, it finds it. Seems to be a problem with the clicking itself, not finding the element. I tried multiple methods. I am trying to debug this for days and I can't figure it out.

This is my code. the site makeup.uk is just an example, my sites are in czech so I didn't want to confuse people.

const puppeteer = require("puppeteer");

(async function main() {
  try {
    const browser = await puppeteer.launch({
      headless: true,
      defaultViewport: {
        width: 1920,
        height: 1080,
      },
    });
    const page = await browser.newPage();

    const response = await page.goto("https://makeup.uk/", {
      waitUntil: ["domcontentloaded", "networkidle2"],
    });

    // first way, doesn't work
    const xp = '::-p-xpath("//button[contains(text(), "Accept")]")';
    const el = await page.waitForSelector(xp);
    await el.click();

    // second way, doesn't work
    const buttons = await page.$$('::-p-xpath("//button")');
    for (const button of buttons) {
      let text = await button.evaluate((el) => el.textContent);
      if (text.includes("Accept")) {
        console.log("Found my button!"); //This logs in the console!
        await button.evaluate((b) => b.click());
      }
    }

const date = new Date().toTimeString();
    await page.screenshot({ path: date + ".png", fullPage: true });
    await browser.close();
  }

    catch (err) {
      console.error(err);
    }
  })();

If I log the el from the first way in the terminal it says

el CdpElementHandle {
  handle: CdpJSHandle {},
  [Symbol(_isElementHandle)]: true
}

which I understand as - I found the element.

I tried multiple other ways like find it within page.evaluate(), and I don't even remember what else, the result is the same. Every time I find the button, just the click does nothing. The cookie banner is in the screenshot every time. Same for other sites, not just this one.

My puppeteer version is this (also tried the lastest one) "puppeteer": "^22.4.1"

Solution

Your selectors work OK for me, although XPath is discouraged. '::-p-xpath("//button")' can be simply 'button' as a CSS selector. See this post for better text selection in Puppeteer.

Upon dismissal, the banner animates gradually down, so if it doesn't complete in time for the screenshot, you'll still see it.

In browser automation outside of testing environments, there's no obligation to act like a human would. A non-animated way to get rid of the banner is to simply blast it out of the DOM instantly with JS:

const puppeteer = require("puppeteer"); // ^22.7.1

const url = "<Your URL>";

(async function main() {
  let browser;

  try {
    browser = await puppeteer.launch({
      defaultViewport: {
        width: 1920,
        height: 1080,
      },
    });
    const [page] = await browser.pages();
    const response = await page.goto(url, {waitUntil: "networkidle2"});
    const banner = await page.waitForSelector(".cookie-notice");
    await banner.evaluate(el => el.remove());
    const path = new Date().toISOString() + ".png";
    await page.screenshot({path, fullPage: true});
  } catch (err) {
    console.error(err);
  }
  finally {
    await browser?.close();
  }
})();

On other sites, remember that the banner may be in an iframe or shadow root, so this isn't expected to be a silver bullet technique that will work on any banner.

In cases where .remove() doesn't fire the needed events, and you're forced to click a button, you can use page.waitForFunction to poll until the element is removed from the DOM or hidden as another way to avoid taking the screenshot while the animation is still in progress after the click.

Other notes about your code:

["domcontentloaded", "networkidle2"] can be simply "networkidle2". "domcontentloaded" is guaranteed to resolve by the time "networkidle2" does.
Generally avoid element handles unless you need trusted events for testing. For scraping, just use an untrusted DOM click, which is more reliable than a trusted click.

If this doesn't answer your question, I suggest sharing your actual site, since its specific behavior may matter.

Note also that cookie banner clicking is often unnecessary for scraping data--you can usually pull the data without worrying about visibility by focusing on untrusted DOM events and network request interception. If your actual use case is a screenshot, though, then you're doing the right thing here.