Search code examples
ruby-on-railsrubyphantomjswatir-webdriver

Watir-webdriver not grabbing the specified divs, and not sure why?


So this is the code for the controller, that I'm trying to use.

class ScraperController < ApplicationController

  def getinformation
      require 'watir-webdriver'
      require 'phantomjs'

      @browser = Watir::Browser.new:phantomjs
      @browser.goto "http://reddit.com"
      @divs = @browser.divs(class: "title may-blank ")

  end

end

On the view page, I have <%= @divs.length %> and it's outputting 0. When I type <%= browser.title %> on the view page it gives me "reddit: the front page of the internet", so I know I'm getting something from the webpage.

I found the div class "title may-blank " by inspecting the titles of some of the posts on the home page, and I have tried using the div class "title may-blank " and "title may-blank". One with the space at the end, how it's shown when it's inspected on the reddit page and one without the space at the end. Both give me nothing though.


Solution

  • Assuming you are trying to locate:

    <a class="title may-blank outbound" data-event-action="title" href="https://i.redd.it/kxtpiskh9mez.jpg" tabindex="1" data-href-url="https://i.redd.it/kxtpiskh9mez.jpg" data-outbound-url="https://out.reddit.com/t3_6si8wj?url=https%3A%2F%2Fi.redd.it%2Fkxtpiskh9mez.jpg&amp;token=AQAAKQiLWXNK30ZkH9-TplVYPCVEIJwNWgqFZCdFuOc3O5l1c7M1&amp;app_name=reddit.com" data-outbound-expiration="1502283817000" rel="">Fries</a>
    

    The problem is the element is an a rather than a div. You need to do at least:

    @browser.links(class: "title may-blank ")
    

    While this will return results, it might not be everything you expect. Supporting multiple classes in a String is also being deprecated in Watir.

    I think you actually want just:

    @browser.links(class: "title")
    

    If you do want both classes, use:

    @browser.links(class: ["title", "may-blank"])