Kotlin: Convert binary InputStream to base64 string in blocks / buffered?

Question

I currently try to read a given inputStream as base64 encoded string in Kotlin. Sadly, it crashes the whole Android application with out of memory error as soon as the file is bigger than around 20 MegaBytes. I like to extend this for at least 40MB.

I think I need to read and convert the bytes of the stream in blocks like 8K for processing, but I can't find any Kotlin example about how to do this as I cannot simply convert the blocks to base64 and concatenate them. This does not work for base64 because of the encoding.

I think I need some sort of stream base64 encoder and I can't find any solution for this that works for me. The JAVA code I can find here in StackOverflow I'm not able to convert to Kotlin as I'm new to Kotlin and Java in general :-(

Also, I'm reading binary data and can't utilize any lineReader function I find every here and there. It should work on blocks with given byte size.

This is the function that currently crashes if the file is bigger than about 20MB:

private fun pushAttachmentToJS(uri: Uri) {
    try {
        val inputStream = contentResolver.openInputStream(uri)
        inputStream.use { stream ->
            
            // Next two lines need a replacement working in blocks
            val fileBytes = stream?.readBytes()
            val fB64 = Base64.encodeToString(fileBytes, Base64.NO_WRAP) // <-- CRASH

            // (... further processing of the fB64 string)
        }
    } catch (e: IOException) {
        e.printStackTrace()
    }
}

Is someone having something that works in Kotlin?

EDIT: I had some partial success in the meantime. But still issues I have to verify...

private fun pushAttachmentToJS(uri: Uri) {
    try {
        val inputStream = contentResolver.openInputStream(uri)
        inputStream.use { stream ->
            
            val encoder = Base64.getEncoder().withoutPadding()
            val buffer = ByteArray(12000) // bytes buffer (multiple of 3!)
            val builder = StringBuilder()

            var bytesRead: Int
            var lastSize = 0
            while (true) {
                bytesRead = stream!!.read(buffer)
                if (bytesRead == -1) break
                lastSize = bytesRead // remember to later get the padding

                val chunk = buffer.copyOfRange(0, bytesRead)
                val encodedChunk = encoder.encodeToString(chunk)

                // Append the encoded chunk to the StringBuilder
                builder.append(encodedChunk)
            }

            // Add necessary padding to the final string
            val remainder = lastSize % 4
            if (remainder != 0) {
                val paddingNeeded = 4 - remainder
                repeat(paddingNeeded) {
                    builder.append('=')
                }
            }

            val fB64 = builder.toString()

            // (... further processing of the fB64 string)
        }
    } catch (e: IOException) {
        e.printStackTrace()
    }
}

But now the Base64 encoded string seems not okay. At least, it does not work as intended. Will need further inspection tomorrow...

EDIT: I found a solution. See answer post below.

Answer 1

I finally solved it this way to save as much memory as possible:

Encode Uri to base64 string:

private fun convertUriToB64(uri: Uri) {
    try {
        val inputStream = contentResolver.openInputStream(uri)
        inputStream?.use { stream ->
            val filename = getFileNameFromUri(uri)
            val encoder = Base64.getEncoder().withoutPadding()
            val buffer = ByteArray(480000) // bytes buffer (multiple of 3!)
            val builder = StringBuilder()
            var bytesRead: Int
            var sLength = 0
            while (true) {
                bytesRead = stream.read(buffer)
                if (bytesRead == -1) break

                val chunk = buffer.copyOfRange(0, bytesRead)
                val encodedChunk = encoder.encodeToString(chunk)
                builder.append(encodedChunk)
                // count b64 string length to later get padding remainder
                sLength += encodedChunk.length
            }

            // Add padding
            val remainder = sLength % 4
            if (remainder != 0) {
                val paddingNeeded = 4 - remainder
                repeat(paddingNeeded) {
                    builder.append('=')
                }
            }

            // handle the result here:
            return builder.toString()
        }
    } catch (e: IOException) {
        e.printStackTrace()
    }
}

Decode base64 string to file:

@OptIn(ExperimentalEncodingApi::class)
fun storeB64asFile(storageFilePath: File, base64binary: String): Boolean {
    val step = 40000 // multiple of 4!
    var pos = 0
    val os = FileOutputStream(storageFilePath)
    while (true) {
        var end = pos + step
        if (end > base64binary.length) {
            end = base64binary.length
        }
        os.write(kotlin.io.encoding.Base64.decode(base64binary, pos, end))
        pos += step
        if (pos > base64binary.length) {
            break;
        }
    }

    os.flush()
    os.close()

    return true
}

This is stripped down. You might need to add additional error handling and input validation.