Trying to get the "what" form (aka job title), from indeed.com
Error when trying to run the program:
/var/lib/gems/2.3.0/gems/capybara-2.11.0/lib/capybara/node/finders.rb:44:in `block in find': Unable to find field "q" (Capybara::ElementNotFound)
Inspecting element via firefox from indeed.com yields: name="q"
<span class="inwrap">
<input class="input_text" maxlength="512" size="31" aria-labelledby="what_label_top hidden_colon what_label_bot" name="q" autocomplete="off" id="what">
</span>
<div style="width:250px"><!-- --></div>
Which matches the code in the scraper:
def perform_search
# For indeed
fill_in 'q', :with => @skillset
fill_in 'l', :with => @region
find('#fj').click
sleep(1)
end
The entire code can be found at: https://github.com/jasnow/job-hunter/blob/master/scraper.rb
Now the problem here is the inability to locate name="q" Are there any other ways I could link to that form on indeed.com so I could initiate webscraping? I'm talking xpath or css perhaps.
Your code only allows the URL http://www.indeed.com , but that URL redirects to https://www.indeed.com and also hits http://indeed.com. Therefore your page load is being blocked. Change to config.allow_url("indeed.com") and it should be able to find the input.