Search code examples
rubystripe-paymentsmechanize-ruby

Net::HTTPNotFound for https://dashboard.stripe.com/login -- unhandled response

We need to retrieve access to the logs for our Stripe instance for a specific time period. There isn't an endpoint in there API (grrrr) so we are trying a quick screen scrape, because the dashboard structures them quite nicely.

At this point though I can't even log into Stripe using Mechanize. Below is the code I am using to log in

require 'mechanize'

agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
agent.follow_meta_refresh = true

starting_link = 'https://dashboard.stripe.com/login'

page = agent.get(starting_link)
login_form = page.form

login_form.email = email
login_form.password = pass

new_page = agent.submit(login_form, login_form.buttons[0])

The response I get from running this is:

Mechanize::ResponseCodeError: 404 => Net::HTTPNotFound for      https://dashboard.stripe.com/login -- unhandled response
from /Users/Nicholas/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.4/lib/mechanize/http/agent.rb:316:in `fetch'
from /Users/Nicholas/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.4/lib/mechanize.rb:1323:in `post_form'
from /Users/Nicholas/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.4/lib/mechanize.rb:584:in `submit'
from (irb):21
from /Users/Nicholas/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'

I tried logging into several other sites and was successful. I also aliased the agent and handled the re-direct (a strategy mentioned in other questions).

Does anyone know what tweaks could be made to Mechanize log into Stripe?

Thanks much

Solution

  • Short Answer:

    I would suggest using a browser engine like Selenium to get the logs data as that will be much simpler.

    Long Answer:

    Though your mechanize form submission code is correct, it is assuming the Stripe login form is being submitted using a normal POST request which is not the case.

    The Stripe login form is being submitted using an AJAX request.

    Here is the working code to take that into account:

    require 'mechanize'

agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
agent.follow_meta_refresh = true

starting_link = 'https://dashboard.stripe.com/login'

page = agent.get(starting_link)
login_form = page.form

login_form.action = 'https://dashboard.stripe.com/ajax/sessions'
login_form.email = email
login_form.password = password

new_page = agent.submit(login_form, login_form.buttons[0])

    As you can see, simply setting the form's action property to the AJAX url solves your problem.

    However, once you have logged in successfully, navigating around the site to scrape it for logs will not be possible with mechanize as it does not support javascript. You can check that by requesting the dashboard's url. You will get an error message to enable javascript.

    Further, Stripe's dashboard is fully javascript powered. It simply makes an AJAX request to fetch data from the server and then render it as HTML.

    This can work for you as the server response is JSON. You can simply parse it and get the required information from logs.

    Upon further inspection(in Chrome Developer Tools), I found that the logs are requested from the url https://dashboard.stripe.com/ajax/logs?count=5&include%5B%5D=total_count&limit=10&method=not_get

    Again, if you try to access this url using mechanize, you will run into CSRF token problem which is maintained between requests by Stripe.

    The CSRF token problem can be solved using mechanize cookies but it will not be worth the effort.

    I would suggest using a browser engine like Selenium to get the logs data as that will much simpler.