Search code examples
rubyweb-scrapingrubygemsnokogirimechanize

How to scrape all reviews if they are on different pages?

How to scrape all reviews from walmart review page (ex:http://www.walmart.com/ip/Ematic-9-Dual-Screen-Portable-DVD-Player-with-Dual-DVD-Players-ED929D/28806789) if they are on different pages?I scrape by mechanize(nokogiri) but it can't click on button(it is not part of form,then I can't submit it)

<button class="paginator-btn paginator-btn-next"><span 
class="visuallyhidden">Next Page</span></button>

and I can't go to next page.How to solve this problem?

Solution

  • I solve this task with watir gem.Mechanize cant interact with JavaScript.