Search code examples
javaweb-scrapingjsouphtmlunit

HtmlUnit Failed to fetch Page - javascript block

First I would like to mention that I am newbie to HtmlUnit. I am trying to fetch a webpage and than parse its content. URL: https://secure.fattal.co.il/BE_Results.aspx?Lang=heb&In=2016-08-07&Out=2016-08-11&Region=2&Rooms=1&Ad1=2&Ch1=0&Inf1=0

but I ended up with the javascript content below:

<html><head><meta charset="utf-8"></head><body><script>window.rbzns = {fiftyeightkb: 43200000, days_in_week : 1};</script><script src="//d1a702rd0dylue.cloudfront.net/js/sugarman/v7/flat.js"></script><script>rbzns.challdomain=".fattal.co.il"; rbzns.ctrbg="NMdHaCamRWvRkFxSyeq856yW5EEmzuHN32UH9RoO3YoeT4HIU++m8k1QIpK0EQqM2RF/9vvBg5S4A3I18QPa4mRMEb+S4Fh3ZVljis2xiCe2tYg/zlUJMN5kVMgLQKw/mbEk8L77gcYBrz56tLIPxg==";rbzns.rbzreqid="fc167e1a3134363136353639343033970795be87d166"; winsocks(true);</script></body></html>

Note: I started to fetch the URL using jsoup parser, but the response was the same javascript.

After googling, on the javascript content, I realize that I should use a headless browser like HtmlUnit.

But even when using HtmlUnit to fetch the page, the javascript content is back again.

Below is a fraction of:

String url = "https://secure.fattal.co.il/BE_Results.aspx?Lang=heb&In=2016-08-07&Out=2016-08-11&Region=2&Rooms=1&Ad1=2&Ch1=0&Inf1=0"; // hard coded just for example
webClient = new WebClient();

// Get the first page
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);

webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setRedirectEnabled(true);
webClient.setJavaScriptTimeout(100000);
webClient.waitForBackgroundJavaScript(100000);

page = webClient.getPage(url);
System.out.println(page.getWebResponse().getContentAsString());

I search all over for a solution, without any concrete answer.

Any help will be appreciated

Solution

  • Ok, I figure it out.

    The javascript source I post:

    <html><head><meta charset="utf-8"></head><body><script>window.rbzns = {fiftyeightkb: 43200000, days_in_week : 1};</script><script src="//d1a702rd0dylue.cloudfront.net/js/sugarman/v7/flat.js"></script><script>rbzns.challdomain=".fattal.co.il"; rbzns.ctrbg="NMdHaCamRWvRkFxSyeq856yW5EEmzuHN32UH9RoO3YoeT4HIU++m8k1QIpK0EQqM2RF/9vvBg5S4A3I18QPa4mRMEb+S4Fh3ZVljis2xiCe2tYg/zlUJMN5kVMgLQKw/mbEk8L77gcYBrz56tLIPxg==";rbzns.rbzreqid="fc167e1a3134363136353639343033970795be87d166"; winsocks(true);</script></body></html>

    is a known script for checking if the request came from a browser and not some sort of bot or crawler. Since HtmlUnit supports(partly) JavaScript, I have change my code to work with Selenium.webdriver + phantomJS, and its now work like a charm.