Search code examples
javascriptjavaparsingjsouphtmlunit

How to trigger jQuery script on site by Java parser


I'm trying to parse a vacancies from https://www.epam.com/careers/job-listings?query=java&department=all&city=Kyiv&country=Ukraine
But I dont get anything execept plain text like "Job Listings Global/English Deutschland/Deutsch Россия/Русский"
The problem is when you load a page - browser runs a script that load some vacancies, but how can I undesrstand JSOUP cant "simulate" browser and run a script. I tried HtmlUnit, but it also done nothing.
Question: What should i do? Am I doing something wrong with HtmlUnit?

Jsoup

Element page = = Jsoup.connect("https://www.epam.com/careers/job-listings?sort=best_match&query=java&department=all&city=all&country=Poland").get();

HtmlUnit

try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_52)) {
    page = webClient.getPage("https://www.epam.com/careers/job-listings?query=java&department=all&city=Kyiv&country=Ukraine");
}

I think i need manualy run some script with

result = page.executeJavaScript("function aa()");

But which one?


Solution

  • You just need to wait a little as hinted here.

    You can use:

    try (final WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
        String url = "https://www.epam.com/careers/job-listings?query=java&department=all&city=Kyiv&country=Ukraine";
    
        HtmlPage page = webClient.getPage(url);
        Thread.sleep(3_000);
        System.out.println(page.asXml());
    }