Search code examples
rubyparsingnokogiricurb

Can't collect pagination

while page <= last_page

  pagination=("https://www.petsonic.com/snacks-huesos-para-perros/?=#{page}")
  puts pagination
  doc2=Nokogiri::HTML(Curl.get(pagination).body_str)

  links=doc2.xpath('//a[@class="product-name"]/@href')
  links.each do |url|
    doc3=Nokogiri::HTML(Curl.get(url).body_str)
    name1=doc3.xpath('//h1[@class="product_main_name"]').text
    puts name1
  end
  page +=1
end

However,for some reason it collects information from first page 4 times, instead of collection from all 4 pages. What might be the issue?

EDIT: i missed 'p' in pagination=("https://www.petsonic.com/snacks-huesos-para-perros/?p=#{page}")

Case closed

Solution

  • It is a simple typo. Just replace

    pagination=("https://www.petsonic.com/snacks-huesos-para-perros/?=#{page}")

    with

    pagination=("https://www.petsonic.com/snacks-huesos-para-perros/?p=#{page}")

    Note the p in the query params.