Search code examples
arraysrubyweb-scrapingiteration

How to iterate and scrape data in Ruby?


I am pretty new to programming and need some help/feedback on my code. My goal is to scrape my data, which is working fine, and then display that data to my user in a numbered list. I am simply having difficulty displaying this data. I do not get any errors back my program simply skips my method altogether. Thanks in advance for any help/feedback!

class BestPlaces::Places
  attr_accessor :name, :population, :places
    @@places = []

  def self.list_places
    # puts "this is inside list places"
    self.scrape_places
  end

      def self.scrape_places
        doc = Nokogiri::HTML(open("https://nomadlist.com/best-cities-to-live"))
            places = doc.search("div.text h2.itemName").text
            rank = doc.search("div.rank").text

            places.collect{|e| e.text.strip}
              puts "you are now in title"
              @@places << self.scrape_places
              puts "#{rank}. #{places}"
            end
          end
        end

CLI Page:
class BestPlaces::CLI

  def list_places
    puts "Welcome to the best places on Earth!"
    puts @places = BestPlaces::Places.list_places
  end

  def call
    list_places
    menu
    goodbye
  end
end

Solution

  • There are a few things that could be addressed in this code, but let's first see a reworking:

    require 'nokogiri'
    require 'open-uri'
    
    module BestPlaces
    
      class Places
        attr_accessor :name, :population, :places
    
        def initialize
          @places = []
        end
    
        def scrape_places
          doc = Nokogiri::HTML(open("https://nomadlist.com/best-cities-to-live"))
          places = doc.search("div.text h2.itemName")
          ranks = doc.search("div.rank")
          places.each{|e| @places << e.text.strip}
          puts "you are now in title"
          @places.each do |place|
            i = @places.index(place)
            puts "#{ranks[i].text}. #{place}"
          end
       end
    
     end
    
     class CLI
    
       def list_places
         puts "Welcome to the best places on Earth!"
         BestPlaces::Places.scrape_places
       end
    
       def call
         list_places
         menu
         goodbye
       end
    
     end
    
    end
    

    You have what looks to be an incomplete Module/Class setup. One can call the above like so:

    bp = BestPlaces::Places.new
    bp.scrape_places
    

    The @@places variable was unnecessary, we can use @places instead to hold values that need to be accessed within the Places class. Also, nokogiri returns a string object when using the .text method on search results, which means you cannot iterate over them like an array. I hope this helps.