Search code examples
javajsoup

Java Jsoup parsing a site but not getting the needed elements


enter image description here

https://goworkabit.com/tooampsud

How to parse this element's information? I have tried this:

try {
    public static final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
    Connection connection; = Jsoup.connect("https://goworkabit.com/tooampsud").userAgent(USER_AGENT);
    Document doc; = connection.get();
    Elements jobs = doc.select("c-workbites-list__workbites-title");
    for (int i = 0; i < jobs.size(); i++) {
            System.out.println(jobs.get(i).text());
    }
}  

Solution

  • Since the data in the table is loaded using javascript, when you just parse the page using Jsoup it will not contains any data from the table. One options is to render the page using any page rendering tools like Selenium+web browser and then parse the page using Jsoup

    You can attempt something like the following code. This should give you the required data to further parse. Please note this approach is quite slow as it expects a physical browser to load the data.

            WebDriver driver = new ChromeDriver();
            try {
                driver.get("https://goworkabit.com/tooampsud");
                Document doc = Jsoup.parse(driver.getPageSource());
                Elements jobs = doc.select("tr.c-workbites-table__workbites-row");
                for (Element job : jobs) {
                        System.out.print(job.select("td>a.c-workbites-table__workbites-title").text() +"    ,   ");
                        System.out.println(job.select("td.text-right").text());
                }
            } catch(Exception e){
                e.printStackTrace();
            } finally {
                driver.quit();
            }