Why does crawling relatedwords.org shows no results?

Question

I'm trying to crawl relatedwords.org website for similar words but the blocks in which the words are being shown are not there in the source code. Here's the code I'm using:

public class SimilarWords {

public static void main(String[] args) throws IOException {

    Document homePage = Jsoup.connect("https://relatedwords.org/relatedto/towing").ignoreHttpErrors(true).get();

    System.out.println(homePage);

}

}

What might be the reason for it?

Answer 1

You are basicly requesting the wrong url.

https://relatedwords.org/relatedto/towing returns the same as https://relatedwords.org

If you inspect the requests made by the browser you can see an XHR request which is requesting the actual answers for your searched term: https://relatedwords.org/api/related?term=towing

This is the url you have to query. It returns your data neatly in JSON format.

You can inspect the requests with the developer tool in the browser . Press F12