Search code examples
rubywatir

scrape sector and industry form morningstar

I want to scrape the Sector and Industry from Morningstar page. I can see the data and the Watir is also seeing it. But when I try to grab the div it does not return anything.

   irb(main):001:0> require 'watir'
=> true

irb(main):008:0> browser= Watir::Browser.new

DevTools listening on ws://127.0.0.1:49780/devtools/browser/4e473d9e-4818-45ad-8238-587bc931099a
=> #<Watir::Browser:0x..f0e9773de url="data:," title="">
irb(main):006:0> path="http://quote.morningstar.ca/Quicktakes/stock/stock_beta.aspx?t=GOOG&region=USA&culture=en-CA"
=> "http://quote.morningstar.ca/Quicktakes/stock/stock_beta.aspx?t=GOOG&region=USA&culture=en-CA"
irb(main):007:0> goto(path)
irb(main):009:0> browser.goto(path)
[41088:42292:1007/225520.743:ERROR:platform_sensor_reader_win.cc(242)] NOT IMPLEMENTED
=> "http://quote.morningstar.ca/Quicktakes/stock/stock_beta.aspx?t=GOOG&region=USA&culture=en-CA"
irb(main):010:0> browser.text.include?"Sector"  #### CAN FIND THE word sector.
=> true
irb(main):011:0> browser.div(:class=>"sal-dp-panel")  ##### it cannot find the class at all.
=> #<Watir::Div: located: false; {:class=>"sal-dp-panel", :tag_name=>"div"}>
    irb(main):015:0> divs=browser.divs(:class=>"sal-dp-panel")
=> #<Watir::DivCollection:0x000000079722d0 @query_scope=#<Watir::Browser:0xdbd2266a url="http://quote.morningstar.ca/Quicktakes/stock/stock_beta.aspx?t=GOOG&region=USA&culture=en-CA" title="GOOG 1157.35 -0.93 (Alphabet Inc Class C)">, @selector={:class=>"sal-dp-panel", :tag_name=>"div"}>
irb(main):018:0> divs.count
=> 0
irb(main):019:0> divs.each{|div| puts div.text}
=> []
irb(main):020:0> divs.each{|div| puts "got one"}
=> []

enter image description here

Solution

  • The problem is that there are no elements on the page with the class "sal-dp-panel". Maybe you meant to get the "sal-dp-pair", which is the div containing the name/value pair?

    <div class="sal-dp-pair">
  <div class="sal-dp-name ng-binding">Sector</div>
  <div class="sal-dp-value ng-binding">Technology</div>
</div>

    To scrape the sector and industry, you can find the relevant "sal-dp-name" and then find it's corresponding value (ie following sibling):

    browser.div(class: 'sal-dp-name', text: 'Sector').following_sibling.text
#=> "Technology"

browser.div(class: 'sal-dp-name', text: 'Industry').following_sibling.text
#=> Internet Content & Information"