Extract page content where there is a delay in the actual content using Playwright

Question

I am trying to capture the privacy notice of this page - "https://www.imaginegolf.com/privacy". However, if you look at the page - it takes a while to load the privacy notice. Is there a way to make playwright wait and grab the contents of the page? I tried options like load, networkidle, commit and domcontentloaded

Sample source code

import {chromium}  from 'playwright'; // Web scraper Library
import * as fs from 'fs';

(async function () {
    const chromeBrowser = await chromium.launch({ headless: true }); // Chromium launch and options
    const context = await chromeBrowser.newContext({ ignoreHTTPSErrors: true ,
        userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
      });
    const page = await context.newPage();
    await page.goto("https://www.imaginegolf.com/privacy", { waitUntil: 'networkidle', timeout: 60000 });
    let content = await page.content();
    fs.writeFileSync('test.html', content);
    console.log("done")
})();

Answer 1

The best way is to probably wait for an element that contains or is part of the content, basically something that indicates the content you expect is loaded. Since you’re just using Playwright Library (not Test), you can’t use expect to asynchronously expect it to be visible within a time frame, but you’re not trying to assert it anyways, just wait for it before moving on. So I would recommend using the waitFor method, like so:

await page.getByText(‘Privacy Notice’).waitFor()

Or with whatever locator best makes sense. Note that waitFor defaults to waiting until it is visible, hence being able to not pass any arguments, though you can always be explicit if you want.