Search code examples
ajaxjsoup

how to find out AJAX network call url populating particular field on a webpage?


On the page https://www.moneycontrol.com/india/stockpricequote/banks-private-sector/karurvysyabank/KVB

TTM PE data gets fetched a bit slower and till that time data is shown as "--" instead.

Questions are in this case

  • why does Jsoup.connect returns document and
  • how to find out which network call url is fetching this particular data ?
  • Is there a way to make jsoup connect wait for all the network calls are complete ?

Solution

  • The page that you see is a dynamic page - some of it is static HTML and some of it is been fetched after the static part has been loaded. While loading it, the HTML part uses "--" just as a placeholder, until the "real" data will arrive.
    You cannot set a delay to wait until all the data will be fetched, since jsoup will not load it anyway - it can handle only the static HTML part of the page.
    In order to find the network call that contains your particular data - most of the odds are that it is some json file, so open the browser's dev tools (F12), go to the network tab and look at the jsons. You'll have to open each of the files and look for the data. In your case I think that the file is https://priceapi.moneycontrol.com/pricefeed/nse/equitycash/KVB (it's called PE). You'll have to fetch it with some other library, like apache httpClient, and parse the json with another library, like jackson.