CasperJS/SpookyJS css selector is existing and not-existing

Question

I have a strange problem during screen scraping with spookyjs / capserjs.

I want to catch information from the following website: 'https://www.rwe-smarthome.de/is-bin/INTERSHOP.enfinity/WFS/RWEEffizienz-SmartHome-Site/de_DE/-/EUR/ViewApplication-DisplayWelcomePage'.

Because the site contains more than one page of products I want to open the other sites too.

Normally one could use

this.click(selector, function() {}); 

to achieve this.

For some strange reasons, this is not working here.

Please have a look at the following code:

var selector1 = "div#workingTemplate div:first-of-type ul.linkList li:nth-child(2) a";

spooky.waitUntilVisible(selector1);
spooky.thenClick(selector1);
spooky.wait(500);
spooky.then(function() {
    this.capture("RWETest-02.jpg");
});

I receive an error

CasperError: Cannot dispatch mousedown event on nonexistent selector: div#workingTemplate div:first-of-type ul.linkList li:nth-child(2) a

Which is strange because if the selector / DOM object does not exists, it should fail at waitUntilVisible().

Also when I try to check if the selector exists, the answer seems to be yes because I also get the error with nonexistent selector:

Code:

spooky.then([{sel: selector1},function() {
    if(this.exists(sel)) {
        this.click(sel);
        this.wait(500);
        this.then(function() {
            this.capture("RWETest-02.jpg");
        });
    }
    else {
        this.emit("logMessage", "Selector does not exists...");
    }
}]);

Error:

CasperError: Cannot dispatch mousedown event on nonexistent selector: div#workingTemplate div:first-of-type ul.linkList li:nth-child(2) a

Because of SpookyJS I use PhantomJS 1.9.7 and CasperJS 1.1.0-beta3.

Does anyone have an idea about this?

Answer 1

This is very likely related to a bug in PhantomJS 1.x which doesn't correctly find elements based on CSS selectors that use :nth-child(). See this question for more information.

Since CasperJS supports XPath expressions for almost all of its functions, you can translate the CSS selector to an XPath expression:

var xpathExpr1 = "//div[@id='workingTemplate']//div[1]//ul[contains(@class,'linkList')]//li[2]//a";

You can then use it like this:

var selectXPath = 'xPath = function(expression) {
  return {
    type: "xpath",
    path: expression,
    toString: function() {
      return this.type + " selector: " + this.path;
    }
  };
};'
...
spooky.then([{x: selectXPath}, function() {
    eval(x);
    this.waitUntilVisible(xPath(xpathExpr1));
    this.thenClick(xPath(xpathExpr1));
    ...
]);

The problem is that SpookyJS doesn't expose the XPath utility, so you need to do a little workaround that is described in GitHub isse #109.