Search code examples
javaweb-scrapingjsouphtmlunit

Scrape/extract with Java, result from coinmarketcap.com

I need to extract coinmarket cap volume (ex: Market Cap: $306,020,249,332) from top of page with Java, please see picture attached.

enter image description here

I have used jsoup library in Java Eclipse but didn't extract volume. Jsoup extract only other attributes. Probably problem is from a java script library.

Also I have used html unit without success:

import java.io.IOException;
import java.util.List;

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class Testss {
    public static void main(String\[\] args) throws IOException {

        String url = "https://coinmarketcap.com/faq/";

        WebClient client = new WebClient();
        HtmlPage page = client.getPage(url);
        List<?> anchors = page.getByXPath("//div\[@class='col-sm-6 text-center'\]//a");
        for (Object obj : anchors) {
            HtmlAnchor a = (HtmlAnchor) obj;
            System.out.println(a.getTextContent().trim());
        }
    }
}

How can I extract volume from this site with Java?

Thanks!

Solution

  • Check the network tab findout the exact request which is fetching the data, In your case its https://files.coinmarketcap.com/generated/stats/global.json

    enter image description here

    Also the request URL is the below oneenter image description here

    So, Fetching the main URL will not give you what you require, For that you have to fetch the data from the request URL directly and parse it using any JSON library. SimpleJSON I can suggest in one of those.

    The JSON data which you will get after hitting the url.

    {
  "bitcoin_percentage_of_market_cap": 55.95083004655126,
  "active_cryptocurrencies": 1324,
  "total_volume_usd": 21503093761,
  "active_markets": 7009,
  "total_market_cap_by_available_supply_usd": 301100436864
}