I’m going to preface this by saying I’m a relative beginner, and while I may not understand a lot yet, I am more than willing to learn. I hope you can help with this one, as I have been trying to figure this out for a long time and am still stuck. I know this is a long one - I wouldn't throw it out here unless I had already put in many days of trying to figure it out myself first.
I am using Mechanize to run a script in my Rails app to fill out several webforms. It worked perfectly locally. However, I need to use delayed_job in order to run the time-intensive script in the background. For now, I am developing this locally (it was previously on Heroku) until the problem is fixed.
So I installed the delayed_job_active_record gem, and have been trying to finagle this into working. I DO know that although my form successfully takes in all of parameters successfully, nothing is getting inserted into the SQL table. It’s not picking up on the object’s ID. It's not recognizing the object's class (it sees it as NilClass), so it considers even regular Ruby methods (such as .strip) as unknown. (The console's output showing all of this is toward the bottom).
I’m clearly doing something wrong. It could very well be something painfully obvious - but I don't have the skills yet to figure that out. I’d be forever indebted to you if you can help me figure out what I’m doing wrong here.
First, here is my code:
MODEL:
require 'digest'
class Jumper < ActiveRecord::Base
after_save do |runscript|
runscript.delay.scrape
end
validates :myuserid, :presence => true
validates :mypass, :presence => true
validates :mydate, :presence => true, :numericality => true
validates :mymonth, :presence => true, :numericality => true
validates :myyear, :presence => true, :numericality => true
validates :mylist, :presence => true
attr_accessor :myuserid, :mypass, :mydate, :mymonth, :myyear, :mylist
def scrape
agent = Mechanize.new
page = agent.get('http://mywebsite.org/')
myform = page.form_with(:name => 'signinForm')
myuserid_field = myform.field_with(:name => "email")
myuserid_field.value = myuserid
mypass_field = myform.field_with(:name => "password")
mypass_field.value = mypass
page = agent.submit(myform, myform.buttons.first)
mylistarray = mylist.strip.split(/[\s]+/)
mylistfinal = mylistarray.map{|l| l[0..-5].sub(/(.*)\./,'\1').gsub('.','/')}.uniq
mylistfinal.each do |doi|
url ='http://mywebsite=' + doi + '&role=some_info#some_dates'
page = agent.get("http://mywebsite.org/submit")
page = agent.get("#{url}")
entryform = page.form_with(:name => 'submit')
entryform.field_with(:name => 'month').options[("#{mymonth}").to_i].select
entryform.field_with(:name => 'day').options[("#{mydate}").to_i].select
entryform.field_with(:name => 'year').options[("#{myyear}").to_i].select
page = agent.submit(entryform, entryform.button_with(:name => 'continue'))
end
end
end`
CONTROLLER:
def index
@doilists = Doilist.all
respond_to do |format|
format.html # index.html.erb
format.json { render json: @doilists }
end
end
def show
@doilist = Doilist.find(params[:id])
respond_to do |format|
format.html # show.html.erb
format.json { render json: @doilist }
end
end
def new
@doilist = Doilist.new
respond_to do |format|
format.html # new.html.erb
format.json { render json: @doilist }
end
end
def create
@jumper = Jumper.new(params[:jumper])
if @jumper.save
flash[:notice] = "Sucessfully submitted your information."
redirect_to @jumper
else
render :action => 'new'
end
end`
GEMFILE:
source 'https://rubygems.org'
gem 'rails'
group :development do
gem 'sqlite3', '1.3.5'
end
group :assets do
gem 'sass-rails', '~> 3.2.5'
gem 'coffee-rails', '~> 3.2.2'
gem 'uglifier', '>= 1.2.3'
end
gem 'jquery-rails', '2.0.2'
gem 'mechanize'
group :production do
gem 'pg', '0.12.2'
end
gem 'delayed_job_active_record'
gem 'daemons'
gem 'thin'`
PROCFILE:
web: bundle exec rails server thin -p $PORT -e $RACK_ENV
worker: bundle exec rake jobs:work`
script/delayed_job:
require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'delayed/command'
Delayed::Command.new(ARGV).daemonize
RAILS_ENV=production script/delayed_job start --exit-on-complete`
Here is the output when I run Foreman in my terminal:
11:33:30 web.1 | Started POST "/MyJumper" for 127.0.0.1 at 2013-05-02 11:33:29 -0400
11:33:30 web.1 | Delayed::Backend::ActiveRecord::Job Load (1.2ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE ((run_at <= '2013-05-02 15:33:30.437882' AND (locked_at IS NULL OR locked_at < '2013-05-02 11:33:30.438065') OR locked_by = 'delayed_job host:myroot pid:7081') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5
1:33:30 web.1 | Processing by JumpersController#create as HTML
11:33:30 web.1 | Parameters: {"utf8"=>"✓", "jumper"=>{"myuserid"=>"[email protected]", "mypass"=>"mypassword123”, "mylist"=>"listitem", "mymonth"=>"4", "mydate"=>"30", "myyear"=>"1"}, "commit"=>"Submit Your Entry"}
11:33:30 web.1 | (0.1ms) begin transaction
11:33:30 web.1 | SQL (21.4ms) INSERT INTO "jumpers" ("created_at", "mydate", "mylist", "mymonth", "mypass", "myuserid", "myyear", "updated_at") VALUES (?, ?, ?, ?, ?, ?, ?, ?) [["created_at", Thu, 02 May 2013 15:33:30 UTC +00:00], ["mydate", nil], ["mylist", nil], ["mymonth", nil], ["mypass", nil], ["myuserid", nil], ["myyear", nil], ["updated_at", Thu, 02 May 2013 15:33:30 UTC +00:00]]
11:33:30 web.1 | SQL (0.5ms) INSERT INTO "delayed_jobs" ("attempts", "created_at", "failed_at", "handler", "last_error", "locked_at", "locked_by", "priority", "queue", "run_at", "updated_at") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [["attempts", 0], ["created_at", Thu, 02 May 2013 15:33:30 UTC +00:00], ["failed_at", nil], ["handler", "--- !ruby/object:Delayed::PerformableMethod\nobject: !ruby/ActiveRecord:Jumper\n attributes:\n id: 68\n mylist: \n created_at: 2013-05-02 15:33:30.920857000 Z\n updated_at: 2013-05-02
15:33:30.920857000 Z\n myuserid: \n mypass: \n mymonth: \n mydate: \n myyear: \nmethod_name: :scrape\nargs:\n- :jumper\n"], ["last_error", nil], ["locked_at", nil], ["locked_by", nil], ["priority", 0], ["queue", nil], ["run_at", Thu, 02 May 2013 15:33:30 UTC +00:00], ["updated_at", Thu, 02 May 2013 15:33:30 UTC +00:00]]
11:33:30 web.1 | (10.7ms) commit transaction
11:33:30 web.1 | Redirected to `http://mylocalhost:5000/MyJumper/68`
11:33:30 web.1 | Completed 302 Found in 74ms (ActiveRecord: 33.1ms)
11:33:31 web.1 | Started GET "/show/68" for 127.0.0.1 at 2013-05-02 11:33:30 -0400
11:33:31 web.1 | Processing by JumpersController#show as HTML
11:33:31 web.1 | Parameters: {"id"=>"68"}
11:33:31 web.1 | Jumper Load (0.3ms) SELECT "jumpers".* FROM "jumpers" WHERE "jumpers"."id" = ? LIMIT 1 [["id", "68"]]
11:33:34 worker.1 | [Worker(host:myroot pid:10470)] Jumper#scrape failed with NoMethodError: undefined method `strip' for nil:NilClass - 0 failed attempts
11:33:34 worker.1 | [Worker(host:myroot pid:10470)] 1 jobs processed at 0.8520 j/s, 1 failed ...
Thanks for your suggestions. I ended up figuring out how to solve this on my own. The issue apparently was that Delayed_Job has significant problems with attr_accessor in Rails 3.
There is more information on this here: delayed_job: attribute accessor values are not stored in delayed_jobs table in rails 3 after upgrade from rails 2
And here: YAML, delayed_job : Psych vs Syck. How to make pysch read attr_accessors for a ruby object
And a workaround for this problem is explained here: https://gist.github.com/jrafanie/3011499
After simply removing the attr_accessor part of my model, everything works perfectly. What a relief!
Also: I removed the after_save code from my model and simply added @jumper.delay.scrape in my controller, right after my controller saved my object. This may not be necessary, but it's more consistent with the documentation:
if @jumper.save
@jumper.delay.scrape
flash[:notice] = "Currently sending your info"
redirect_to start_index_path
As I move towards production, I will take a further examination of the workaround and see the best way to my project so I can have attr_accessor in my model. But for now, just glad to have this working!