Search code examples
node.jsweb-scrapingrequestcheerio

Cheerio web scraping ul > li attribute


I want to scrape the "data-price" from this url, but it is coming back undefined. Any ideas?

https://i.sstatic.net/N9gei.png

async function variant() {
    const response = await axios("https://extrabutterny.com/collections/release-draws/products/nike-sb-dunk-low-pro-blue-fury-bq6817-400?variant=31800767021104");   
    console.log("response: ", response);
    const html = await response.data;
    const $ = cheerio.load(html);
    const id = $(".DrawApp-SizeChartList > li").attr("data-price");
    console.log(id);
}

Solution

  • When the HTML is returned by the backend, there is no List item in the class you are looking for. Here's how it looks:

    <ul class="DrawApp-SizeChartList">
    </ul>
    

    There is no list item. That's the reason the id is undefined. The JavaScript runs in the browser and populates the list.

    The good thing is you don't need to run a browser/puppeteer to get the price. The website makes a query to backend to get the details about the product.

    Here's the url: https://eb-draw.herokuapp.com/draws/4482352611376

    You'll see price here:

    "variants": [
      {
        "id": 16793,
        "draw_id": 1764,
        "variant_id": "31800767021104",
        "variant_label": "8",
        "variant_price": "100.00",
        "winner_count": 3,
        "winners_left": 3,
        "current_entries": 0,
        "created_at": "2020-03-27 21:19:14",
        "updated_at": "2020-03-27 21:19:14"
      },
      ...
    ] 
    

    But where do you get the id for the product? It's here:

    $(".js-price-preview").attr("data-product-id")
    // "4482352611376"
    

    Hope this helps!