Search code examples
androidsqliteretrofit2

Inserting + 4500 items with 7 columns each into SQlite Database with Retrofit2


I am working on an Android App for handheld Scan Devices and want to download around 4.500 items from an MySQL database via Retrofit2 into a SQlite Database on the device; hence, when I tried to download all items at once, it slowed down the UI and froze the app up to 5 minutes; I googled a lot on how to solve the problem and couldn´t come up with a fitting solution for now; hence I tried to download the Database with 7 columns for each item - hence, around 31.500 entries in the database - in "Chunks" by iterating in a For-each loop and using .stream() and .limit() in a Background threat, like this:

 public static void writeItemsToDatabase(Context mContext, String basic) {

        //creating the itemApi interface
        ItemApi itemApi = retrofit.create(ItemApi.class);

        //making the call object
        Call<List<Item>> call = itemApi.checkItems(basic);

        call.enqueue(new Callback<List<Item>>() {
            @Override
            public void onResponse(@NonNull Call<List<Item>> call,
                                   @NonNull Response<List<Item>> response) {
                if (response.isSuccess()) {
                    List<Item> itemList;
                    itemList =  response.body();
                    int dataSize = response.body().size();
                    Log.d(TAGGG, String.valueOf(dataSize));
                    itemList.forEach(List -> Log.d(TAGGG, String.valueOf(List.getEan())));
                    itemList.forEach(List -> Log.d(TAGGG, String.valueOf(List.getNo())));
                    class DownloadTask extends AsyncTask<String, Integer, String> {

                        // Runs in UI before background thread is called
                        @Override
                        protected void onPreExecute() {
                            super.onPreExecute();
                            // Do something like display a progress bar
                        }

                        // This is run in a background thread
                        @Override
                        protected String doInBackground(String... params) {
                            // Do something that takes a long time, for example:
                            for (int i = 0; i <= 3 ; i++) {
                                try (DatabaseHandler itemsManager = new DatabaseHandler((XXXXApp)
                                        mContext.getApplicationContext())) {
                                    itemList.stream().limit(1500).forEach(item -> {
                                        itemsManager.addItem(item);
                                        itemsManager.close();
                                    });
                                }
                                // Call this to update your progress
                                publishProgress(i);
                            }
                            return "this string is passed to onPostExecute";
                        }

                        // This is called from background thread but runs in UI
                        @Override
                        protected void onProgressUpdate(Integer... values) {
                            super.onProgressUpdate(values);
                            // Do things like update the progress bar
                        }
                        // This runs in UI when background thread finishes
                        @Override
                        protected void onPostExecute(String result) {
                            super.onPostExecute(result);
                            // Do things like hide the progress bar or change a TextView
                        }
                    }
                    new DownloadTask().execute();
                }
            }

            @Override
            public void onFailure(Call<List<Item>> call, Throwable t) {}
        });
        return;
    }

however, the result is not satisfying as the Database doesn´t get´s downloaded properly; I changed the values for i to 9 and .limit() to 500 (to achieve the same result, the Download of +4.500 Items) with the same result.

The problem certainly is in this code snippet:

for (int i = 0; i <= 3 ; i++) {
     try (DatabaseHandler itemsManager = new DatabaseHandler((XXXApp)
          mContext.getApplicationContext())) 
   {
         itemList.stream().limit(1500).forEach(item -> {
         itemsManager.addItem(item);
         itemsManager.close();
         });
         }
   // Call this to update your progress
publishProgress(i);
                           

}

It is the nearest approach that I´ve found to what I want to achieve after googling a lot; the problem certainly is that it´s a For-Loop that closes the Database each time and reopens it; I am also not sure if the amount of Data is too big for an SQlite database; hence any help or hints on how to solve this properly would be very much appreciated, thanks!


Solution

    1. Create once instance of DatabaseHandler(what is it? you can use room with more comfortable API) and reuse it.
    2. Insert many(100-500) items in one transaction.

    Or you can create sqlite db file on server side then download it and open as DB in android app.