How do you process large amount of data to be render?

Question

In the below code example, 50 thousand Customer needs to be loaded using ListView or RecyclerView

// current process, I am using pagination to load items
List<Customer> customers = mCustomerModule.get(limit, offset);
mAdapter.addItems(customer);
    

• Couchbase lite version 1.3
• The retrieval of customers is made by views

Problem

1. Slow when scrolling, when reached a high amount of customers

Question

1. Does my pagination process slow down the performance?
2. Couchbase lite 1.3 views are really slow?

Goal

1. Improve performance
2. Smoothen viewing of customer list

Answer 1

What you are looking for is called Endless Scrolling. Take a look at this article that describe the problem and present the solution for both: with ListView and with RecyclerView:

Endless Scrolling with AdapterViews and RecyclerView

UPDATE

Here's how to implement it using a RecyclerView (info taken from above link)

Use this code as is that would be the EndlessRecyclerViewScrollListener, that will react to scrolls on the RecyclerView. So you need to set the addOnScrollListener of your RecyclerView to an instance of this class:

public abstract class EndlessRecyclerViewScrollListener extends RecyclerView.OnScrollListener {
// The minimum amount of items to have below your current scroll position
// before loading more.
private int visibleThreshold = 5;
// The current offset index of data you have loaded
private int currentPage = 0;
// The total number of items in the dataset after the last load
private int previousTotalItemCount = 0;
// True if we are still waiting for the last set of data to load.
private boolean loading = true;
// Sets the starting page index
private int startingPageIndex = 0;

RecyclerView.LayoutManager mLayoutManager;

public EndlessRecyclerViewScrollListener(LinearLayoutManager layoutManager) {
    this.mLayoutManager = layoutManager;
}

public EndlessRecyclerViewScrollListener(GridLayoutManager layoutManager) {
    this.mLayoutManager = layoutManager;
    visibleThreshold = visibleThreshold * layoutManager.getSpanCount();
}

public EndlessRecyclerViewScrollListener(StaggeredGridLayoutManager layoutManager) {
    this.mLayoutManager = layoutManager;
    visibleThreshold = visibleThreshold * layoutManager.getSpanCount();
}

public int getLastVisibleItem(int[] lastVisibleItemPositions) {
    int maxSize = 0;
    for (int i = 0; i < lastVisibleItemPositions.length; i++) {
        if (i == 0) {
            maxSize = lastVisibleItemPositions[i];
        }
        else if (lastVisibleItemPositions[i] > maxSize) {
            maxSize = lastVisibleItemPositions[i];
        }
    }
    return maxSize;
}

// This happens many times a second during a scroll, so be wary of the code you place here.
// We are given a few useful parameters to help us work out if we need to load some more data,
// but first we check if we are waiting for the previous load to finish.
@Override
public void onScrolled(RecyclerView view, int dx, int dy) {
    int lastVisibleItemPosition = 0;
    int totalItemCount = mLayoutManager.getItemCount();

    if (mLayoutManager instanceof StaggeredGridLayoutManager) {
        int[] lastVisibleItemPositions = ((StaggeredGridLayoutManager) mLayoutManager).findLastVisibleItemPositions(null);
        // get maximum element within the list
        lastVisibleItemPosition = getLastVisibleItem(lastVisibleItemPositions);
    } else if (mLayoutManager instanceof GridLayoutManager) {
        lastVisibleItemPosition = ((GridLayoutManager) mLayoutManager).findLastVisibleItemPosition();
    } else if (mLayoutManager instanceof LinearLayoutManager) {
        lastVisibleItemPosition = ((LinearLayoutManager) mLayoutManager).findLastVisibleItemPosition();
    } 

    // If the total item count is zero and the previous isn't, assume the
    // list is invalidated and should be reset back to initial state
    if (totalItemCount < previousTotalItemCount) {
        this.currentPage = this.startingPageIndex;
        this.previousTotalItemCount = totalItemCount;
        if (totalItemCount == 0) {
            this.loading = true;
        }
    }
    // If it’s still loading, we check to see if the dataset count has
    // changed, if so we conclude it has finished loading and update the current page
    // number and total item count.
    if (loading && (totalItemCount > previousTotalItemCount)) {
        loading = false;
        previousTotalItemCount = totalItemCount;
    }

    // If it isn’t currently loading, we check to see if we have breached
    // the visibleThreshold and need to reload more data.
    // If we do need to reload some more data, we execute onLoadMore to fetch the data.
    // threshold should reflect how many total columns there are too
    if (!loading && (lastVisibleItemPosition + visibleThreshold) > totalItemCount) {
        currentPage++;
        onLoadMore(currentPage, totalItemCount, view);
        loading = true;
    }
}

// Call this method whenever performing new searches
public void resetState() {
    this.currentPage = this.startingPageIndex;
    this.previousTotalItemCount = 0;
    this.loading = true;
}

// Defines the process for actually loading more data based on page
public abstract void onLoadMore(int page, int totalItemsCount, RecyclerView view);

}

Inside the mentioned onLoadMore method, load additional items into the adapter either by sending out a network request or by loading from another source.

For example:

scrollListener = new EndlessRecyclerViewScrollListener(linearLayoutManager) {
       @Override
       public void onLoadMore(int page, int totalItemsCount, RecyclerView view) {
           // Triggered only when new data needs to be appended to the list
           // Add whatever code is needed to append new items to the bottom of the list
           loadNextDataFromApi(page);
       }
  };
  // Adds the scroll listener to RecyclerView
  rvItems.addOnScrollListener(scrollListener);

When you intend to perform a new search, make sure to clear the existing contents from the list and notify the adapter the contents have changed as soon as possible. Make sure also to reset the state of the EndlessRecyclerViewScrollListener with the resetState method:

// 1. First, clear the array of data
listOfItems.clear();
// 2. Notify the adapter of the update
recyclerAdapterOfItems.notifyDataSetChanged(); // or      notifyItemRangeRemoved
// 3. Reset endless scroll listener when performing a new search
scrollListener.resetState();