Search code examples
google-apps-scriptweb-scraping

How to pull data from a website using Apps Script


Objective:
To pull data from: https://www.olg.ca/en/home.html

LottoMax Jackpot amount & # of max millions
Lotto649 Gold Ball amount
LOTTARIO Jackpot amount

My Problem:
The amounts are dynamically rendered in JavaScript so can't use ImportXML. I created a function in Apps Script, but I don't see an API to fetch the data from, so it won't work.

Is there any other method I can use to fetch this data? Any insight will be greatly appreciated. Thanks.


Solution

  • Fetching dynamic website using Google Sheets or Apps Script

    After spending time trying to scrape the site using apps script and google sheets and also following discussions like this and Stackoverflow posts like this it is safe to say that Google Sheets’ IMPORTXML and even Google App Script’s UrlFetchApp does not have the capability to render dynamically rendered websites.

    To accomplish the goal, you would need to use other tools like Selenium WebDriver that has the capability to wait for the site to load up completely and then scrape it.

    You may also try submitting the idea of fetching dynamic websites using apps script as a feature request here

    References:

    Google Sheets scraping

    How to extract dynamic website data with Google Script?

    Utilizing Selenium WebDriver for Dynamic Web Scraping in C#