Why HtmlUnit code doesn't match Chrome code?

Question

I am trying to web router page scraping to search for connected devices and information about them. I written this code:

String searchUrl="http://192.168.1.1";
HtmlPage page=client.getPage(searchUrl);
System.out.println(page.asXml());

The problem is that the code returns by HtmlUnit is different from the code in Chrome. In HtmlUnit I don't have the section of code that lists connected devices.

Answer 1

try something like this

try (final WebClient webClient = new WebClient()) {
    // do not stop at js errors
    webClient.getOptions().setThrowExceptionOnScriptError(false);

    webClient.getPage(searchUrl);
    // now wait for the js starting async (you can play with the timing)
    webClient.waitForBackgroundJavaScript(10000);

    // maybe the page was replaced from the async js
    HtmlPage page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

    System.out.println(page.asXml());
}

Usually that helps. If you still facing problems you have to open an issue on github (https://github.com/HtmlUnit/htmlunit).

But keep in mind i can only really help if i can run/and debug your code here - means you web app has to be public.