Watir wait for more posts load

Question

The site which I want to scrape is having lazy loading with a scroll event. (infinite scroll) I want to wait for more posts to load, but all posts have the same class name. So, I can't wait for an element with a specific class name to load. I don't want to use sleep() method.

Example Code:

<div class='posts'>
 <div class='single-post'></div>
 <div class='single-post'></div>
 <div class='single-post'></div>
 <div class='single-post'></div>
 // The following posts appear after scrolling. I want to fetch these posts
 <div class='single-post'></div>
 <div class='single-post'></div>
 <div class='single-post'></div>
 <div class='single-post'></div>
</div>

How can I solve this issue? Is it possible to check if this div has more than 10 posts?

Thoughts?

Answer 1

You can get a collection of the single-post divs to see if more have loaded. Depending on the markup of the page, you can do:

browser.divs(class: 'single-post').count

or, if the parent matters:

browser.div(class: 'posts').divs(class: 'single-post').count

This can be used to wait for the number of posts to increase:

# Find the original count
original_count = browser.divs(class: 'single-post').count

# Take an whatever action to trigger the loading of more posts
browser.scroll.to :bottom

# Wait for the count to increase
browser.wait_until do
  new_count = browser.divs(class: 'single-post').count
  new_count > original_count
end