Search code examples
javahtmljsoup

Retrieving a span value in html using jsoup

I am trying to retrieve the number "1" from the "display-ratings" span class in the snippet of html below:

 <article class="surveys-verbatim__item ">
    <h3 id="engine_major">
        <a class="surveys-verbatim__item-title surveys-verbatim__item-title--font-large crux-body-copy crux-body-copy--bold" data-cr-collapse data-target=".surveys-verbatim__item" href="javascript:void(0);">
            Engine Major
            <span class="crux-icons crux-icons-caret-up-small surveys-verbatim__item__arrow"></span>
        </a>
    </h3>
    <div class="surveys-verbatim__item-body">
        <div class="row">
            <div class="col-sm-6 col-xs-12">
                <p class="crux-body-copy crux-body-copy--small surveys-verbatim__item-description">Engine rebuild or replacement, cylinder head, head gasket, turbo or supercharger, timing chain or timing belt.</p>
            </div>
                                <div class="col-md-offset-2 col-md-4 col-sm-offset-1 col-sm-5 col-xs-12">
                    <div class="surveys-verbatim__item-ratings">
                        <div class="brick-chart-chart-container">
    <div class="brick-chart-chart-wrapper-container clearfix">
<div class="brick-chart-chart-wrapper poor">
    <div class="brick-chart-brick"></div>
    <div class="brick-chart-brick active" style="width: 20%"></div>
</div>

<div class="score-range ">
                <span><span class="display-ratings">1</span> / 5</span>
        </div>
</div>
</div>
                    </div>
                </div>
                        </div>
                </div>
</article>

I am using Jsoup in Java and have the following code:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class Main {

public static void main(String[] args) throws Exception {

    final Document document = Jsoup.connect("https://www.consumerreports.org/cars/audi/a6/2006/reliability?pagestop").get();
    for (Element element : document.select("article.surveys-verbatim__item")) {
        String rating = element.select("span.display-ratings").text();
        System.out.println(rating);
    }

  }
}

This output blank strings. Can anyone let me know if they know how to extract the value listed in the span class "display-ratings" ?

Thank you!

Solution

  • The rating you are trying to extract is visible only if you login to the website. Jsoup parsed the URL, but is not able to extract the rating because it is hidden.

    If you want to get the rating then you have to login to the website programmatically and parse the webpage.

    HereHere is a screenshot of the page source before logging in. You can see that rating is locked here i.e, not visible here.