How to sort 144,000 files by lastModified() and length() in Kotlin within a second?

Question

I have the following ArrayList of File.

    var a  = ArrayList<File>()
    var b  = ArrayList<File>()
    var c  = ArrayList<File>()
    var d  = ArrayList<File>()
    var e  = ArrayList<File>()

Once the application has started the above ArrayList will be filed with more than 144,000 files. The total size all these combined would nearly 3.5 GB. I want to sort them by lastModified() or length() within in a second and update the modified ArrayList into RecyclerView.

For ease of sorting I have made above ArrayList into a Array<ArrayList<File>> as follows :

    val mList  = arrayOf(a,b,c,d,e)

To speed up things I do everything in background thread. Code :

          doAsync {
              mList.forEach{ index ->
                  index.sortByDescending { it.lastModified() }
              }
              activityUiThread {
                  setRecyclerViewAdapter() // Update RecyclerView with new sorted files
              }

          }

Link to the library I used sort files do in background thread : https://github.com/Kotlin/anko

The above code takes nearly 3-5 seconds to execute. I want this to be done within a second. How to solve this issue ? I am doing this for android.

If needed Im ready to change the API to do the background task

Answer 1

I want this to be done within a second

Short: it is impossible in general case. File System (on Android or on any other OS) can be overloaded, so this operation can pause your application sometimes. Please note this.

However you can speedup the code by using the following algorithm:

• Read file metadata in parallel
• Sort files via these results.

Please use example below. Please note that it does a lot of parallel IO operations.

Benefits of this solution:

• Metadatas are read from the separate context (which has thread limit, to avoid IO overuse)
• Metadata of file X is read only one time.
• Sorting algorithm works with operating memory only, e.g. it use ready data, which reduces IO access.

suspend fun sortFiles(files: Iterable<File>): List<File> {
    val metadataReadTasks: List<Deferred<FileWithMetadata>> = withContext(Dispatchers.IO) 
 {
        files.map { file ->
            async {
                FileWithMetadata(file)
            }
        }
    }
    val metadatas: List<FileWithMetadata> = metadataReadTasks.awaitAll()
    return metadatas
        .sorted()
        .map {
            it.file
        }
}
private class FileWithMetadata(
    val file: File
) : Comparable<FileWithMetadata> {
    private val lastModified = file.lastModified()
    private val length = file.length()
    override fun compareTo(other: FileWithMetadata): Int {
        return when (other.length) {
            this.length -> other.lastModified.compareTo(this.lastModified)
            else -> other.length.compareTo(this.length)
        }
    }
}