How to grab content from web site like this in Perl

Question

I want to grab data from the web site : http://mops.twse.com.tw/mops/web/t05st03

Set the value of input id "co_id" to 1101

<input id="co_id" class="textbox" type="text" onkeydown="{if(event.keyCode==13){ajax1(document.form1,'table01');}}" size="10" value="" name="co_id"></input>

and then click the button

<input type="button" onclick="javascript:doAction();hideIt2('quicksearch9');ajax1(document.form1,'table01');" value="搜尋"></input>

the web site will show extra data out same web page, that data is what I want to grab.

I wrote a perl code

my $url="http://mops.twse.com.tw/mops/web/t05st03"; 
my $mech = WWW::Mechanize->new( );
$mech->get($url);
my $response;
$mech->field(co_id => 1101);
$mech->click_button(name => " 搜尋 ");
$response = $mech->content();
print $response;

but it does not get the data in $mech->content

How can I solve it?

Answer 1

You just need to emulate JavaScript in your script. I used Firefox's HTTPFox extension to find what info you need to POST:

use WWW::Mechanize;

my $url="http://mops.twse.com.tw/mops/web/t05st03";
my $co_id = 1101;

my $mech = WWW::Mechanize->new();
$mech->agent_alias("Windows IE 6");

$mech->get($url);

$mech->post("http://mops.twse.com.tw/mops/web/ajax_t05st03",
        Content => {

            encodeURIComponent => 1,
            step => 1,
            firstin => 1,
            off => 1,
            keyword4 => "",
            code1 => "",
            TYPEK2 => "",
            checkbtn => "",
            queryName => "co_id",
            TYPEK => "all",
            co_id => $co_id,
        });

my $response;

$response = $mech->content();