Search code examples
rubynokogiriopen-uri

How do I locate the tag containing this text?

I am using Open::URI and Nokogiri to scrape a Google search page:

 require 'open-uri'
 require 'nokogiri'
 url = http://www.google.co.uk/search?&q=toys&start=0&num=&complete=0
 doc = Nokogiri::HTML(open(url))
 mas = doc.css('li.g')[7]
 mas.at_css('.mas-row')

From this result I am interested in just one result:

"Amazon.co.uk: Toys - Harry Potter: Toys & Games"

and I would like to get the data from "div class mas-row".

I can not find it. I looked in the "doc" variable and it can not be found. After that I looked for the text that is in that "div" and for the first div a part of the text was found but nothing from the next div.

Can anyone help me with this?

Solution

  • The div with mas-row is included in the HTML. It's rendered by JavaScript.

    Use a library that can handle JavaScript, such as selenium.