Search code examples
javascriptparsingjsoupphantomjshtmlunit

I can't access the javascript expandable menu at this page with jSoup. How can i make it work using Phantom.js or HTML Unit?

I got a problem when parsing this page. I perfectly get the price, the airline, the departure time, etc.

But what i want is the flight number. And its hidden, you need to click in a flight and the menu expands to show it.

enter image description here Here is where jsoup has to stop. if you click on "view source" anywhere, it won't show e.g. flight "6186". If you expand the menu and rightclick to there, it will give you the flight back.

The specific information I am searching for is this:

enter image description here

So, I noticed I need to use something like a headless browser to access all content. But when trying to understand phantom.js or htmlunit, i got huge problems getting started. I can't formulate what I need and I'm irritated by the features of headless browsers. An example code on how to get this into parseable HTML would be so nice.

Does anyone have experience with parsing elements like these? Thank you so much in advance.

best regards!

UPDATE to answer to jPOs comment. Other help still extremely apreciated - Here is what i see when i inspect the events: enter image description here

UPDATE 2: Any Ideas on how to do this? Maybe headless browsers?

Solution

  • So as an answer look at this code.

    $.ajax({
  url:"http://www.kayak.de/s/run/inlineDetails/flight",
  type:"post",
  dataType:"json",
  data:{
    localidx:104,
    resultid:"ea63237698f20885f47a9b3827f9953b",
    searchid:"kUECCUIWzU"
  }
});

    You can see there is a server waiting for your request. Trying to understand the parameters sent to the server gave me few options, I'm going to stick to these. localidx is the id written in the button Details anzeigen resultid is also written inside that button searchid is written in an iframe with the id master-1 in the name tag

    I hope you are a friend with regexes because that's what you have to face :/

    Good luck! Hope I helped!