I have a project for school to parse web code and use it like a data base. When I tried to down data from (https://www.marathonbet.com/en/betting/Football/), I didn't get it all?
Here is my code:
Document doc = Jsoup.connect("https://www.marathonbet.com/en/betting/Football/").get();
Elements newsHeadlines = doc.select("div#container_EVENTS");
for (Element e: newsHeadlines.select("[id^=container_]")) {
System.out.println(e.select("[class^=block-events-head]").first().text());
System.out.println(e.select("[class^=foot-market]").select("[class^=event]").text());
}
for result you get (this is last of displayed leagues):
Football. Friendlies. Internationals All bets Main bets
1. USA 2. Mexico 16 Apr 01:30 +124 7/5 23/10 111/50 +124
on top of her are all leagues displayed.
Why don't I get full data? Thank you for your time!
Jsoup has a default body response limit of 2MB. You can change it to whatever you need with maxBodySize(int)
Set the maximum bytes to read from the (uncompressed) connection into the body, before the connection is closed, and the input truncated. The default maximum is 2MB. A max size of zero is treated as an infinite amount (bounded only by your patience and the memory available on your machine).
E.g.:
Document doc = Jsoup.get(url).userAgent(ua).maxBodySize(0).get();
You might like to look at the other options in Connection, on how to set request timeouts, the user-agent, etc.