Search code examples
rubynullmechanize-ruby

How do I ignore the nil values in the loop with parsed values from Mechanize?


In my text file are a list of URLs. Using Mechanize I'm using that list to parse out the title and meta description. However, some of those URL pages don't have a meta description which stops my script with a nil error:

undefined method `[]' for nil:NilClass (NoMethodError)

I've read up and seen solutions if I were using Rails, but for Ruby I've only seen reject and compact as possible solutions to ignore nil values. I added compact at the end of the loop, but that doesn't seem to do anything.

require 'rubygems'
require 'mechanize'

File.readlines('parsethis.txt').each do |line|
    page = Mechanize.new.get(line)
    title = page.title
    metadesc = page.at("head meta[name='description']")[:content]
    puts "%s, %s, %s" % [line.chomp, title, metadesc] 
end.compact!

It's just a list of urls in a text like this:

http://www.a.com
http://www.b.com

This is what will output in the console for example:

 http://www.a.com, Title, This is a description.

If within the list of URLs there is no description or title on that particular page, it throws up the nil error. I don't want it to skip any urls, I want it to go through the whole list.


Solution

  • Here is one way to do it:

    Edit( for added requirement to not skip any url's):

    metadesc = page.at("head meta[name='description']")
    puts "%s, %s, %s" % [line.chomp, title, metadesc ? metadesc[:content] : "N/A"]