Search code examples
javascriptcurlwgethttplib2

Command line URL fetch with JavaScript capabliity

I use curl, in php and httplib2 in python to fetch URL.

However, there are some pages that use JavaScript (AJAX) to retrieve the data after you have loaded the page and they just overwrite a specific section of the page afterward.

So, is there any command line utility that can handle JavaScript?

To know what I mean go to: monster.com and try searching for a job.

You'll see that the Ajax is getting the list of jobs afterward. So, if I wanted to pull in the jobs based on my keyword search, I would get the page with no jobs.

But via browser it works.

Solution

  • you can use PhantomJS http://phantomjs.org

    You can use it as below : 

    var page=require("webpage");
page.open("http://monster.com",function(status){
  page.evaluate(function(){
    /* your javascript code here 
        $.ajax("....",function(result){


            phantom.exit(0);
           }); */
  });
});