Search code examples
rubyweb-crawlerdynamic-content

crawling dynamic content using ruby


I am using ruby gems (nokogiri & mechanize) to make a crawler for a website but this website contains bootstrap modals (popup windows) that's generated dynamically on button click.

this content (of modal) shows up on button click that uses a "get" method on some URL.

I am getting the response by crawling the URL associated with the button but I am just getting the same page source.

how could I get the content of that dynamic content using "ruby" ?


Solution

  • That modal you're describing, with high probability is rendered with a Js. So what you're looking for is not possible, because mentioned libs do not execute Js.

    In order to parse pages whose content is Js dependent, you should use other tools, e.g. puppeteer