Search code examples
rubyweb-scrapingwatir

fetching all the data from a partially hidden list watir

I'm trying to fetch a span of 4,600 elements

<span> 4,600 </span>

i inspected the elements and found that each element is a list class which has a child class with a title and href that i want to fetch the problem is that :

  1. not all elements are visible , you would have to scroll down the api to find more elements
  2. i cant seem to successfully fetch a single piece of data

puts browser.th(:class => %w("_9irns _pg23k _jpwof _gvoze")).link.hreflang

this is the structure of the code i'm trying to fetch

<ul class = 'xxx'>
<div class = 'xxa'>
<li class ='fff'>
<li class ='fff'>
<li class ='fff'>
.
.

the <li class = 'fff'> has <a class='xxx xxx xxx xxx'> having the data i'm trying to fetch tittle and href

to be more clear how can I iterate over all the classes of 'fff' and pick a url which is in a child class of it.

Solution

  • Don't use quotes inside the %w to find an element from a collection of classes, and try requiring watigiri gem and using #text! to obtain text of hidden elements.