Search code examples
javascriptnode.jsweb-scrapingchromiumpuppeteer

Puppeteer loads blank page with 429 when accessing URL

I am working on a project to obtain pricing information from a hotel website, but I cannot perform any searches when loading the website in puppeteer.

Here is a snippet of my JavaScript that opens Chrome.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false, devTools: false });
  const page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3683.103 Safari/537.36');

  await page.goto('https://www.hyatt.com/');

})();

How are they detecting that I'm using Chrome controlled by Puppeteer, even though it is running a headful browser?

Thanks

Solution

  • They're fingerprinting you. It's some javascript that enumerates things like your browser plugins and navigator object and sends that back to the server.

    So you'd have to intercept that post and change it to something a normal browser would send.