Search code examples
typescriptplaywrightplaywright-test

How to get all errors and assert them (or add them into report) on the page instead of getting first error and stopping

I am testing html header orders in the following code, derived from this answer. But, test stops when it hits the first error. I want to get all errors on the page instead of getting first error and stops

I need to improve my code for it

async assertHtmlHeadingOrders() {
    await this.page.waitForLoadState('domcontentloaded')
    const failMsg = (prev, curr) =>
      `h${curr} was more than one level deeper than previous header h${prev}`
    const headers = await this.page
      .locator('h1, h2, h3, h4, h5, h6')
      .evaluateAll(els => els.map(el => +el.tagName.slice(1)))
    // eslint-disable-next-line no-plusplus, id-length
    for (let i = 1; i < headers.length; i++) {
      expect(headers[i] - headers[i - 1], failMsg(headers[i - 1], headers[i])).toBeLessThan(2)
    }
  }
Solution

  • You can put the failing pairs into an array, then assert one time on the whole array, enabling a clearer error message.

    import {expect, test} from "@playwright/test"; // ^1.39.0

const failMsg = pairs =>
  `The following header pairs were invalid: ${
    pairs.map(e => `h${e.prev} -> h${e.curr}`).join(", ")}`;

const findInvalidHeaderPairs = async page => {
  const headers = await page
    .locator("h1, h2, h3, h4, h5, h6")
    .evaluateAll(els => els.map(el => +el.tagName.slice(1)));
  return headers
    .slice(1)
    .map((e, i) => ({prev: headers[i], curr: e}))
    .filter(e => e.curr - e.prev > 1);
};

test("fail so the assertion message can be seen", async ({page}) => {
  const html = `<!DOCTYPE html><html><body>
  <h1>1</h1>
  <h2>2</h2>
  <h4>4</h4>
  <h3>3</h3>
  <h5>5</h5>
  <h5>5</h5>
  <h6>6</h6>
  <h3>3</h3>
  <h6>6</h6>
  </body></html>`;
  await page.setContent(html); // replace with page.goto("<Your URL>"); when you're working with a real site
  const pairs = await findInvalidHeaderPairs(page);
  expect(pairs, failMsg(pairs)).toHaveLength(0);
});

    Output:

    $ npx playwright test

Running 1 test using 1 worker

  ✘  1 pw.test.js:17:5 › fail so the assertion message can be seen (98ms)


  1) pw.test.js:17:5 › fail so the assertion message can be seen ───────────────

    Error: The following header pairs were invalid: h2 -> h4, h3 -> h5, h3 -> h6

    expect(received).toHaveLength(expected)

    Expected length: 0
    Received length: 3
    Received array:  [{"curr": 4, "prev": 2}, {"curr": 5, "prev": 3}, {"curr": 6, "prev": 3}]

      30 |   const pairs = await findInvalidHeaderPairs(page);
    > 31 |   expect(pairs, failMsg(pairs)).toHaveLength(0);
         |                                 ^
      32 | });

        at pw.test.js:31:33

  1 failed
    pw.test.js:17:5 › fail so the assertion message can be seen ────────────────

    The same suggestions as in this previous answer apply: adding a custom matcher or polling, as fits your needs.

    Adding a custom matcher would abstract away the logic shown here into the matcher rather than into functions, which is a bit more elegant for the caller. If you're using this on many tests, it's probably worth the extra work. Here's an example that also polls to ensure the test eventually passes:

    import {expect, test} from "@playwright/test";

expect.extend({
  async toHaveValidHeaders(page) {
    let pairs;
    let error;

    try {
      await expect.poll(async () => {
        const headers = await page
          .locator("h1, h2, h3, h4, h5, h6")
          .evaluateAll(els => els.map(el => +el.tagName.slice(1)));
        pairs = headers
          .slice(1)
          .map((e, i) => ({prev: headers[i], curr: e}))
          .filter(e => e.curr - e.prev > 1);
        return pairs;
      }).toHaveLength(0);
    }
    catch (e) {
      error = e;
    }

    if (!pairs) {
      return {
        message: () => error?.message ??
          "unknown page failure in toHaveValidHeaders",
        pass: false,
      };
    }
    else if (pairs.length) {
      return {
        message: () =>
          `The following header pairs were invalid: ${
            pairs.map(e => `h${e.prev} -> h${e.curr}`).join(", ")}`,
        pass: false,
      };
    }

    return {
      message: () => "passed",
      pass: true,
    };
  },
});

test("fail so the assertion message can be seen", async ({page}) => {
  const html = `<!DOCTYPE html><html><body>
  <h1>1</h1>
  <h2>2</h2>
  <h4>4</h4>
  <h3>3</h3>
  <h5>5</h5>
  <h5>5</h5>
  <h6>6</h6>
  <h3>3</h3>
  <h6>6</h6>
  </body></html>`;
  await page.setContent(html);
  await expect(page).toHaveValidHeaders();
});

test("page with valid headers", async ({page}) => {
  await page.goto("https://playwright.dev/docs/test-configuration");
  await expect(page).toHaveValidHeaders();
});

test("page with (eventual) valid headers", async ({page}) => {
  test.setTimeout(10_000);
  const html = `<!DOCTYPE html><html><body>
  <h1>1</h1>
  <h2>2</h2>
  <h4>4</h4>
  <script>
  setTimeout(() => {
    document.querySelector("h4").remove();
  }, 3000);
  </script>
  </body></html>`;
  await page.setContent(html);
  await expect(page).toHaveValidHeaders();
});

    Now the assertion failure is clearer, and we can also work with pages that add more headers after navigation:

    $ npx playwright test

Running 3 tests using 1 worker

  ✘  1 pw.test.js:47:5 › fail so the assertion message can be seen (5.0s)
  ✓  2 pw.test.js:63:5 › page with valid headers (533ms)
  ✓  3 pw.test.js:68:5 › page with (eventual) valid headers (4.0s)


  1) pw.test.js:47:5 › fail so the assertion message can be seen ───────────────

    Error: The following header pairs were invalid: h2 -> h4, h3 -> h5, h3 -> h6

      58 |   </body></html>`;
      59 |   await page.setContent(html);
    > 60 |   await expect(page).toHaveValidHeaders();
         |                      ^
      61 | });
      62 |
      63 | test("page with valid headers", async ({page}) => {

        at pw.test.js:60:22

  1 failed
    pw.test.js:47:5 › fail so the assertion message can be seen ────────────────
  2 passed (10.3s)

    Here's a simpler custom matcher that doesn't include polling, which is better if you only want to check headers at page load, or want to explicitly use a poll in the test:

    import {expect, test} from "@playwright/test";

expect.extend({
  async toHaveValidHeaders(page) {
    const headers = await page
      .locator("h1, h2, h3, h4, h5, h6")
      .evaluateAll(els => els.map(el => +el.tagName.slice(1)));
    const pairs = headers
      .slice(1)
      .map((e, i) => ({prev: headers[i], curr: e}))
      .filter(e => e.curr - e.prev > 1);

    if (pairs.length === 0) {
      return {
        message: () => "passed",
        pass: true,
      };
    }

    return {
      message: () =>
        `The following header pairs were invalid: ${
          pairs.map(e => `h${e.prev} -> h${e.curr}`).join(", ")}`,
      pass: false,
    };
  },
});

test("fail so the assertion message can be seen", async ({page}) => {
  const html = `<!DOCTYPE html><html><body>
  <h1>1</h1>
  <h2>2</h2>
  <h4>4</h4>
  <h3>3</h3>
  <h5>5</h5>
  <h5>5</h5>
  <h6>6</h6>
  <h3>3</h3>
  <h6>6</h6>
  </body></html>`;
  await page.setContent(html);
  await expect(page).toHaveValidHeaders();
});

test("page with valid headers", async ({page}) => {
  await page.goto("https://playwright.dev/docs/test-configuration");
  await expect(page).toHaveValidHeaders();
});

test("page with (eventual) valid headers", async ({page}) => {
  test.setTimeout(10_000);
  const html = `<!DOCTYPE html><html><body>
  <h1>1</h1>
  <h2>2</h2>
  <h4>4</h4>
  <script>
  setTimeout(() => {
    document.querySelector("h4").remove();
  }, 3000);
  </script>
  </body></html>`;
  await page.setContent(html);
  await expect(() => expect(page).toHaveValidHeaders()).toPass();
});