How to parse a hidden Javascript section from a HTML page

I want to parse the dates+prices of the month September from the hidden calendar on this URL: http://www.lufthansa.com/vol/vol-paris-berlin . The problem here is that when you press on the month September the page will generate the calendar but without changement in the url. I used this code but no result.

public static void main(String[] args) 
        throws FailingHttpStatusCodeException, MalformedURLException, IOException {

    WebClient webClient = new WebClient();
    HtmlPage myPage = webClient.getPage("http://www.lufthansa.com/vol/vol-paris-berlin");
    Document doc = Jsoup.parse(myPage.asXml());

    for(Element s : doc.select("button.daygrid_cell.hasprice")) {
        String weekday_text = s.select(".weekday_text").text();
        String pricebox = s.select(".pricebox > .br").text();
        System.out.println(
                String.format(
                        "weekday_text=%s pricebox=%s", 
                        weekday_text, 
                        pricebox));
    }

    webClient.close();}

Solution

I currently don't see a way with htmlUnit myself.

You could "cut out the middleman" though, using the same query the lufthansa page uses to populate the calendar view:

https://bestprice-live-backend.mcon.net/flights-by-day?l=fr_fr&departure=PAR&destination=BER&departureFrom=2016-09-01&departureTo=2016-09-30&cabin=Economy&duration=7

The response is in JSON format, so you could extract the information in the same presentation as on the lufthansa page (prices are always rounded up to next integer) using a JSON parser. In the following example I used json-simple:

Map<String, Integer> prices = new TreeMap<String, Integer>(); // sorted map/keys in sorted order

try {
    Document doc = Jsoup
                .connect("https://bestprice-live-backend.mcon.net/flights-by-day?l=fr_fr&departure=PAR&destination=BER&departureFrom=2016-09-01&departureTo=2016-09-30&cabin=Economy&duration=7")
                .userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36")
                .referrer("http://www.lufthansa.com/vol/vol-paris-berlin")
                .get();

    JSONObject obj = (JSONObject) new JSONParser().parse(doc.text());

    obj = (JSONObject) obj.get("dates");

    for (Iterator<?> iterator = obj.keySet().iterator(); iterator.hasNext();) {
        String key = (String) iterator.next();
        JSONObject dateObject = (JSONObject) obj.get(key);
        Double price = (Double) dateObject.get("price");
        int roundedPrice = (int) Math.ceil(price); // lufthansa displays prices rounded up
        prices.put(key, roundedPrice);
    }

    for (String key : prices.keySet()) {
        System.out.println(key + ": " + prices.get(key) + " €");
    }
} catch (IOException e) {
    e.printStackTrace();
} catch (ParseException e) {
    e.printStackTrace();
}

Output:

2016-09-01: 163 €
2016-09-02: 158 €
2016-09-03: 160 €
2016-09-04: 160 €
2016-09-05: 160 €
2016-09-06: 158 €
2016-09-07: 155 €
2016-09-08: 159 €
2016-09-09: 160 €
2016-09-10: 156 €
2016-09-11: 160 €
2016-09-12: 159 €
2016-09-13: 157 €
2016-09-14: 158 €
2016-09-15: 160 €
2016-09-16: 184 €
2016-09-17: 156 €
2016-09-18: 160 €
2016-09-19: 179 €
2016-09-20: 159 €
2016-09-21: 163 €
2016-09-22: 180 €
2016-09-23: 188 €
2016-09-24: 160 €
2016-09-25: 160 €
2016-09-26: 160 €
2016-09-27: 154 €
2016-09-28: 157 €
2016-09-29: 159 €
2016-09-30: 163 €