HtmlUnit unable to click on spans in Selenium, Java

Question

I'm writing an automation tool in Java using Selenium. The following code works fine if the WebDriver is FirefoxDriver, however if I try to substitute it with HtmlUnitDriver it will crash on the line when it is supposed to click on a span. I have tried making it locate the span both by CSS selector as well as XPath. Making the driver wait until element is loaded with

wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath(".//*[@id='home']/div[3]/div[3]/div[10]/div[3]/div[2]/div[1]/a/span")));

also doesn't solve the issue. The code is:

WebDriver driver = new HtmlUnitDriver(BrowserVersion.FIREFOX_52);

//driver.setJavascriptEnabled(true);

WebDriverWait wait = new WebDriverWait(driver, 10);

//logs in
driver.get("https://www.tribalwars.net/");
driver.findElement(By.id("user")).clear();
driver.findElement(By.id("user")).sendKeys(username);
driver.findElement(By.cssSelector("div.right.login > div.wrap")).click();
driver.findElement(By.id("password")).clear();
driver.findElement(By.id("password")).sendKeys(password);
driver.findElement(By.cssSelector("a.btn-login")).click();

//It executes fine up until this point

driver.findElement(By.cssSelector("span.world_button_active")).click();
/*
This won't work either    
driver.findElement(By.xpath(".//*[@id='home']/div[3]/div[3]/div[10]/div[3]/div[2]/div[1]/a/span")).click();
*/

The error I'm given:

Exception in thread "main" org.openqa.selenium.NoSuchElementException: Returned node was not a DOM element
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'PC', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.0-53-generic', java.version: '1.8.0_131'
Driver info: driver.version: HtmlUnitDriver
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElementByCssSelector(HtmlUnitDriver.java:1247)
    at org.openqa.selenium.By$ByCssSelector.findElement(By.java:430)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver$5.call(HtmlUnitDriver.java:1965)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver$5.call(HtmlUnitDriver.java:1)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.implicitlyWaitFor(HtmlUnitDriver.java:1601)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:1961)
    at org.openqa.selenium.htmlunit.HtmlUnitDriver.findElement(HtmlUnitDriver.java:756)
    at Bot.Test.build(Test.java:86)
    at Bot.Test.main(Test.java:42)

Answer 1

The root cause is your HtmlUnitDriver didn't execute JavaScript. You have to turn it on.

However, the driver will crash because the web site has some errors. So you have to customize HtmlUnitDriver to allow those errors.

What you need to do is inherit the HtmlUnitDriver and override modifyWebClient to turn off some error throws.

import java.util.logging.Level;
import java.util.logging.Logger;

import org.openqa.selenium.htmlunit.HtmlUnitDriver;

import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.WebClient;

public class MyHtmlUnitDriver extends HtmlUnitDriver {

    public MyHtmlUnitDriver(BrowserVersion version, boolean enableJavascript) {
        super(version, enableJavascript);
    }

    protected WebClient modifyWebClient(WebClient client) {
        WebClient modifiedClient = super.modifyWebClient(client);
        Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
        modifiedClient.getOptions().setThrowExceptionOnScriptError(false);
        modifiedClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
        modifiedClient.getOptions().setPrintContentOnFailingStatusCode(false);
        return modifiedClient;
    }
}

Code below shows an example.

public void test() {
    // Use a custom HtmlUnitDriver
    WebDriver hd = new MyHtmlUnitDriver(BrowserVersion.FIREFOX_52, true);
    WebDriverWait wait = new WebDriverWait(hd, 10);

    hd.get("https://www.tribalwars.net/");
    hd.findElement(By.id("user")).clear();
    hd.findElement(By.id("user")).sendKeys(username);
    hd.findElement(By.cssSelector("div.right.login > div.wrap")).click();
    hd.findElement(By.id("password")).clear();
    hd.findElement(By.id("password")).sendKeys(password);
    hd.findElement(By.cssSelector("a.btn-login")).click();

    // Add Wait here 
    wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("span.world_button_active")));
    hd.findElement(By.cssSelector("span.world_button_active")).click();

P.S. I think you should go with PhantomJS or SlimmerJS instead.