Search code examples
node.jsweb-scrapingrequest

Node can't scrape certain pages

I don't know if this is something to do with coldfusion pages or what but I can't scrape these .cfm pages

In the command line in a directory with request run:

node> var request = require('request');
node> var url = 'http://linguistlist.org/callconf/browse-conf-action.cfm?confid=173395';
node> request(url, function (err, res, body) { if (err) { console.log(err) } else { console.log('body:', body) }; });

I've tried with some other .cfm sites but they work, and am only getting blank results so I don't know what it could be

Note: I've also tried doing it the barebones require('http').get(url,…) route but I get the same blank result

Solution

  • I GOT IT ! (finally)

    This web server really need to know how to answer to you. Try this (it work for me)

    var request = require('request');
var options = {
  url: 'http://linguistlist.org/callconf/browse-conf-action.cfm?confid=173395',
  headers: {
   'Accept-Encoding':'none'
  }
};
request(options, function (err, res, body) { if (err) { console.log(err) } else { console.log('body:', body) }; });