Failed to retrieve bundle prices for a steam game with Puppeteer

Question

I'm writing a code to get the title and price of a game's bundles, separating the title and price for each bundle, but it returns that bundle_price is null.

import puppeteer from "puppeteer";

async function handleAgeRestriction(p) {
  await p.evaluate(() => {
    const select = document.querySelector("#ageYear");
    const options = select.querySelectorAll("option");
    const selectedOption = [...options].find(
      (option) => option.text === "1900"
    );

    selectedOption.selected = true;
  });
  await p.click("#view_product_page_btn");
}

async function getDataFromGame() {
  const browser = await puppeteer.launch({ headless: false });

  const page = await browser.newPage();

  await page.goto("https://store.steampowered.com/app/271590/");

  await handleAgeRestriction(page);

  await page.waitForSelector(".apphub_AppName");
  // await page.waitForSelector(".game_purchase_price");
  await page.waitForSelector("div.game_area_purchase_game > h1");
  await page.waitForSelector("div.discount_final_price");

  const result = await page.evaluate(() => {
    const data = document.querySelectorAll('.game_area_purchase_game');
    const game = [...data].map((bundle) => {
      const bundle_title = bundle.querySelector('div.game_area_purchase_game > h1').innerText;
      const bundle_price = bundle.querySelector("div.discount_final_price").innerText;

      return {
        bundle_title,
        bundle_price,
      }
    })
    return game;
  });

  console.log(result);

  await browser.close();
}

getDataFromGame();

I find the error strange because if I replace "bundle" with "document", it correctly retrieves the price, but it will always be the same price

 const result = await page.evaluate(() => {
    const data = document.querySelectorAll('.game_area_purchase_game');
    const game = [...data].map((bundle) => {
      const bundle_title = bundle.querySelector('div.game_area_purchase_game > h1').innerText;
      const bundle_price = document.querySelector("div.discount_final_price").innerText;

      return {
        bundle_title,
        bundle_price,
      }
    })
    return game;
  });

  console.log(result);

(This is an example of how it does NOT throw me an error, but the price will be the same for all bundles.)

Answer 1

The reason for the error (Error [TypeError]: Cannot read properties of null (reading 'textContent')) is that some bundles don't have .discount_final_price elements. Instead, they have a dropdown that lets you select one of a few options with different prices.

Instead of assuming that .discount_final_price is always present, you can use a condition to handle the possibility that it isn't and grab the dropdown list:

const result = await page.$$eval(
  ".game_area_purchase_game",
  els =>
    els.map(bundle => {
      const bundle_title = bundle
        .querySelector("h1")
        ?.textContent.trim();
      const bundle_price = bundle.querySelector(
        ".discount_final_price"
      )?.textContent;

      if (bundle_price) {
        return {
          bundle_title,
          bundle_price,
        };
      }

      bundle
        .querySelector(
          ".game_area_purchase_game_dropdown_selection"
        )
        .click();
      return {
        bundle_title,
        bundle_prices: [
          ...bundle.querySelectorAll(
            ".game_area_purchase_game_dropdown_menu_item_text"
          ),
        ].map(e => e.textContent)
      };
    })
);

Output will now look something like:

[
  {
    bundle_title: 'Buy Shark Cash Cards',
    bundle_prices: [
      'Tiger Shark: GTA$250,000 - $4.99',
      'Bull Shark: GTA$600,000 - $9.99',
      'Great White Shark: GTA$1,500,000 - $19.99',
      'Whale Shark: GTA$4,250,000 - $49.99',
      'Megalodon Shark: GTA$10,000,000 - $99.99'
    ]
  },
  {
    bundle_title: 'Buy Grand Theft Auto V: Premium Edition & Great White Shark Card Bundle',
    bundle_price: '$19.80'
  },
  {
    bundle_title: 'Buy Grand Theft Auto V: Premium Edition & Megalodon Shark Card Bundle',
    bundle_price: '$36.40'
  },
  {
    bundle_title: 'Buy Grand Theft Auto V: Premium Edition',
    bundle_price: '$29.98'
  }
]

You could process this list further and use e.textContent.split(" - ").at(-1) to extract the price from each dropdown item.