Search code examples
ruby-on-railsrubypostscreen-scrapingmechanize

POST request capture with Mechanize


I'm trying to use Mechanize to capture a POST request which is not possible via a form, because the form is inside an iframe that prevents loading directly via javascript.

The HTTP Headers are the following in an example request from Google Chrome (notice the parada and linea parameters)

Request URL:http://www.etr.gov.ar/getSmsResponse.php
Request Method:POST
Status Code:200 OK
Request Headers
Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:es-419,es;q=0.8
Connection:keep-alive
Content-Length:21
Content-Type:application/x-www-form-urlencoded
Host:www.etr.gov.ar
Origin:http://www.etr.gov.ar
Referer:http://www.etr.gov.ar/cont-cuandollega.php
User-Agent:Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.30 Safari/534.30
X-Requested-With:XMLHttpRequest

Form Dataview URL
parada:4152
linea:112

Response Headers
Connection:close
Content-Length:111
Content-Type:text/html
Date:Fri, 03 Jun 2011 02:35:45 GMT
Server:Microsoft-IIS/7.5
X-Powered-By:PHP/5.1.2
ASP.NETl

And the content for this example is:

Linea 112N: 0min. 379mts., siguiente 25min. 9937mts. - Linea 112R: 81min. 24349mts., siguiente 101min. 30548mts

What I've tried with mechanize is the following ruby script, but I get a blank page in response:

require 'mechanize'
agent = WWW::Mechanize.new
agent.post("http://www.etr.gov.ar/getSmsResponse.php", "parada" => "4152", "linea"=>"112")

What could I'd be doing wrong? Thank you very much.

UPDATE: Passing the POST as a hash worked perfectly. To render the content then I only had to do agent.post.content


Solution

  • The post method expects the parameters as a hash. Try:

    agent.post("http://www.etr.gov.ar/getSmsResponse.php", {"parada" => "4152", "linea"=>"112"})