Search code examples
javascriptjavaweb-scrapingjsoup

Jsoup is unable to fetch complete content from a webpage (No errors/exceptions, but misses some content)

I am trying to fetch content from following page with JSOUP:

http://www.exchangeandmart.co.uk/used-cars-for-sale

But it does not fetch the div with id=results, even though it is visible when I open the same link from the browser. Please help me

Java code:

Connection connection = Jsoup.connect("http://www.exchangeandmart.co.uk/used-cars-for-sale");
Document doc = connection.get();
System.out.println(doc.getElementById("results")); // prints null

NOTE: There are no exceptions or errors while downloading the page. Only some of the content in the page is missing. I printed the whole document on console with System.out.println(doc); , it was quite different from the page I view in the browser.

Solution

  • Document doc = Jsoup.connect("http://www.exchangeandmart.co.uk/used-cars-for-sale").userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2").get();

    Now the page should be rendered as accessed from Chrome on PC.