How to get big response body Jsoup

Question

Tell me what the problem is, and how to solve it. I need to get a response array json format. On the server, the response is formed in about 30 minutes, collects an array of 150K elements. Then it comes to dispatch, and hangs on hold for more than 24 hours. But even after such a long time, the answer does not come. There are no errors on the server. If you take 10K, then the answer comes without problems. I tried to save the answer to a TXT file, its size is +-35MB, and contains 35K stock.

This is how I execute the request:

Connection.Response response = Jsoup.connect(URLs.sayt + "/" + URLs.api + "/hs/CLIENTS/LIST/")
                    .method(Connection.Method.GET)
                    .maxBodySize(0)
                    .timeout(0)
                    .userAgent(URLs.getUserAgentMobile())
                    .header("Authorization", "Basic " + base64login)
                    .header("Accept-Language", "*")
                    .header("mode", "tobase")
                    .execute();
            if(response != null){
                ArrayList<Client> clients = new Gson().fromJson(response.body(), new TypeToken<List<Client>>() {}.getType());
                for (Client client:clients){
                    sqLiteWork.AddToClient(client);
                }
            }

Here is an example of an element from the json array:

{"id": "9812d904-2d8a-11e8-80bb-a45d36c35311", "name": "Afaq", "code": "", "isGroup": false, "parent": "null", "address": "", "phone": "+994(12) 436 71 88", "phoneMore": "+994(50)2409696", "phoneHome": "", "debt": 0 }

Answer 1

I haven't found a solution to my question. I've tried both okhttp and httpurlconnection. Apparently, the android client itself does not support such a large string response in the body. Maybe it would be a solution via a file. But this is not a solution when you need actual data. Therefore, I made a response from the server for 10K elements. Full loading of 150K elements takes 2 hours.

Please note, I'm talking up-to-date data, if 2 hours is a long time, then online mode is also supported in my application.

Connection connection = Jsoup.connect(URLs.sayt + "/" + URLs.api + "/hs/CLIENTS/LIST/")
                    .method(Connection.Method.GET)
                    .ignoreContentType(true)
                    .maxBodySize(0)
                    .timeout(0)
                    .userAgent(URLs.getUserAgentMobile())
                    .header("Authorization", "Basic " + base64login)
                    .header("Accept-Language", "*")
                    .header("mode", "tobase");

            Connection.Response response = connection.execute();
            ArrayList<Client> clients = new Gson().fromJson(response.body(), new TypeToken<List<Client>>() {}.getType());
            Client last = null;
            for (Client client:clients){
                last = client;
                sqLiteWork.AddToClient(client);
            }

            while (clients.size() > 0){
                response = connection.header("last", last.id).execute();
                clients = new Gson().fromJson(response.body(), new TypeToken<List<Client>>() {}.getType());
                for (Client client:clients){
                    last = client;
                    sqLiteWork.AddToClient(client);
                }
            }