Login to amazon partnernet using Ruby Mechanize

I try to login into amazon partnernet, e.g. https://partnernet.amazon.de/ using the Ruby Mechanize gem:

Gemfile:

# https://github.com/sparklemotion/mechanize
gem 'mechanize'

The code below is a rake task. It worked in the past, I think Amazon changed the page html so this code is no longer working, e.g. by changing the submit button of the form name="sign_in" to an image type=input.

desc "Cron Task for Email Notifications"
task :email_amazon_stats => :environment do
  puts "Start: Fetch and send Amazon Sales from yesterday (#{Time.now})"

  # login to Amazon Partnernet
  a = Mechanize.new
  a.user_agent_alias = 'Mac Safari'
  a.follow_meta_refresh = true
  a.redirect_ok = true

  a.get('https://partnernet.amazon.de/') do |page|
    # Submit the login form
    page.form_with(:name => 'sign_in') do |f|
      username_field = f.field_with(:id => "username")
      username_field.value = "[email protected]"
      password_field = f.field_with(:id => "password")
      password_field.value = "somepassword"
    end.submit

    start_date  = Time.now - 1.day
    end_date    = Time.now
    my_page2 = a.get("https://partnernet.amazon.de/gp/associates/network/reports/report.html?ie=UTF8&deviceType=all&endDay=#{(end_date.strftime('%d').to_i).to_s}&endMonth=#{((end_date.strftime('%m').to_i)-1).to_s}&endYear=#{end_date.strftime('%Y').to_i.to_s}&periodType=exact&preSelectedPeriod=monthToDate&program=all&reportType=earningsReport&startDay=#{start_date.strftime('%d').to_i.to_s}&startMonth=#{((start_date.strftime('%m').to_i)-1).to_s}&startYear=#{start_date.strftime('%Y').to_s}")

    form = my_page2.form_with(:name => 'htmlReport')
    button = form.button_with(:name => 'submit.download_XML')
    xml = a.submit(form, button)

    # ASIN="3423347570"
    # Binding="paperback"
    # Category="14"
    # Date="December 01, 2015"
    # DeviceType="BROWSER"
    # EDate="1448928000"
    # Earnings="0,65"
    # LinkType="asn"
    # Price="9,25"
    # Qty="1"
    # Rate="7,03"
    # Revenue="9,25"
    # Seller="Amazon.de"
    # Tag="yx-21"
    # Title="Kopf schl&#228;gt Kapital: Die ganz andere Art, ein Unternehmen zu gr&#252;nden Von der Lust, ein Entrepreneur zu sein (dtv Sachbuch)"/>

    doc = Nokogiri::XML(xml.body)
    @sales = []
    doc.xpath("//Item").each do |item|
      @sales << {
          :sale_itemasin    => item['ASIN'],
          :sale_itemname    => item['Title'].truncate(80),
          :sale_date        => Time.at(item['EDate'].to_i).strftime("%Y-%m-%d %H:%M:%S").to_s,
          :sale_amount      => '%.2f' % item['Revenue'].gsub(',','.').to_f,
          :sale_commission  => '%.2f' % item['Earnings'].gsub(',','.').to_f
      }
    end

    earnings = 0
    @sales.each do |s|
      earnings += s[:sale_commission].to_f
    end
    @total_commission = '%.2f' % earnings
  end

  ReportsMailer.daily_dashboard(@total_commission,@sales).deliver
  puts "Done: Fetch and send Amazon Sales from yesterday (#{Time.now})"
end

Can someone help me in this?

I looked for similar questions how to restructure the submit, but so far nothing works. Login is not happening. (Yes, PWD is correct :-) )

Similar question, but does not solve the problem above: Cannot Login to Amazon with Ruby Mechanize

Solution

So.. I debugged the code, this version is working now as expected:

desc "Cron Task for Email Notifications"
task :email_amazon_stats => :environment do
  puts "Start: Fetch and send Amazon Sales from yesterday (#{Time.now})"

  agent = Mechanize.new
  agent.cookie_jar.clear!
  agent.user_agent_alias = 'Mac Firefox'
  agent.follow_meta_refresh = true
  agent.redirect_ok = true

  dashboard_url = "https://partnernet.amazon.de/gp/associates/network/reports/report.html?__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&tag=&reportType=earningsReport&program=all&deviceType=all&periodType=preSelected&preSelectedPeriod=yesterday&startDay=1&startMonth=11&startYear=2016&endDay=2&endMonth=11&endYear=2016&submit.display.x=87&submit.display.y=16&submit.display=Auf+der+Seite+anzeigen"

  agent.get(dashboard_url)

  form = agent.page.form_with(:name => 'sign_in')
  form.username = ENV['AZON_PARTNER_USR']
  form.password = ENV['AZON_PARTNER_KEY']
  form.submit

  dashboard = agent.get(dashboard_url)
  form2 = dashboard.form_with(:name => 'htmlReport')
  button = form2.button_with(:name => 'submit.download_XML')
  xml = agent.submit(form2, button)

  doc = Nokogiri::XML(xml.body)
  @sales = []
  doc.xpath("//Item").each do |item|
    @sales << {
        :sale_itemasin    => item['ASIN'],
        :sale_itemname    => item['Title'].truncate(80),
        :sale_date        => Time.at(item['EDate'].to_i).strftime("%Y-%m-%d %H:%M:%S").to_s,
        :sale_amount      => '%.2f' % item['Revenue'].gsub(',','.').to_f,
        :sale_commission  => '%.2f' % item['Earnings'].gsub(',','.').to_f
    }
  end

  earnings = 0
  @sales.each do |s|
    earnings += s[:sale_commission].to_f
  end

  @total_commission = '%.2f' % earnings

  ReportsMailer.daily_dashboard(@total_commission,@sales).deliver
  puts "Done: Fetch and send Amazon Sales from yesterday (#{Time.now})"
end

As you can see yourself this is pretty ugly, because I try to go to the deeplink directly, which redirects me to the login page. There I login and try again to go to the dashboard. This time it works. Why ugly? Because if I try to go to the login page directly the code does not work, I somehow need this redirect. Any idea why? Would be interesting to understand this...