Search code examples
javascriptnode.jsweb-scrapingphantomjscasperjs

CasperJS continuous run


I have a simple CasperJs script, which looks something like this:

var casper = require('casper').create();

casper.start('http://casperjs.org/', function() {
    this.echo(this.getTitle());
});

casper.thenOpen('http://phantomjs.org', function() {
    this.echo(this.getTitle());
});

casper.run();

Now, I would like this script to execute in loop until I stop it, hence I tried:

var casper = require('casper').create();

setInterval(function() {
   casper.start('http://casperjs.org/', function() {
    this.echo(this.getTitle());
});

casper.thenOpen('http://phantomjs.org', function() {
    this.echo(this.getTitle());
});

casper.run();}, 5000);

But the problem is, casper.run() also exits the file execution, without creating a new casper instance. An ugly workaround for my issue, was just creating a bash script which executed my command in a loop. My question is: how can I execute a command in the terminal casperjs myscript.js which will run my scraping script continuously, with a sleep of x seconds, till I stop it?


Solution

  • No time to test the following but I have done similar thing before:

    casper.start();
    openPages();
    
    function openPages() {
        casper.thenOpen('http://casperjs.org', function() {
            this.echo('First Page: ' + this.getTitle());
        });
    
        casper.thenOpen('http://phantomjs.org', function() {
            this.echo('Second Page: ' + this.getTitle());
        });
    
        openPages();
    }
    

    Edit:

    Use casper.wait() if you want to wait for certain seconds before the next crawl.

    casper.wait(5000, function() {
        openPages();
    });