How to stream zipped file (on the fly) via Play Framework 2.5 in scala?

Question

I want to stream some files and zip them on the fly, so users can download multiple files into a single zipped file without writing anything to the local disk. However, my current implementation holds everything in the memory, and will no work for large files. Is there any way to fix it?

I was looking at this implementation: https://gist.github.com/kirked/03c7f111de0e9a1f74377bf95d3f0f60, but couldn't figure out how to use it.

 import java.io.{BufferedOutputStream, ByteArrayInputStream, ByteArrayOutputStream}
import java.util.zip.{ZipEntry, ZipOutputStream}
import akka.stream.scaladsl.{StreamConverters}
import org.apache.commons.io.FileUtils
import play.api.mvc.{Action, Controller}

class HomeController extends Controller {
  def single() = Action {
                         Ok.sendFile(
                           content = new java.io.File("C:\\Users\\a.csv"),
                           fileName = _ => "a.csv"
                         )
                       }

  def zip() = Action {
                     Ok.chunked(StreamConverters.fromInputStream(fileByteData)).withHeaders(
                       CONTENT_TYPE -> "application/zip",
                       CONTENT_DISPOSITION -> s"attachment; filename = test.zip"
                     )
                   }

  def fileByteData(): ByteArrayInputStream = {
    val fileList = List(
      new java.io.File("C:\\Users\\a.csv"),
      new java.io.File("C:\\Users\\b.csv")
    )

    val baos = new ByteArrayOutputStream()
    val zos = new ZipOutputStream(new BufferedOutputStream(baos))

    try {
      fileList.map(file => {
        zos.putNextEntry(new ZipEntry(file.toPath.getFileName.toString))
        zos.write(FileUtils.readFileToByteArray(file))
        zos.closeEntry()
      })
    } finally {
      zos.close()
    }

    new ByteArrayInputStream(baos.toByteArray)
  }
}

Answer 1

Instead of using a ByteArrayOutputStream to buffer the contents in an array then putting them into a ByteArrayInputStream you could use Java's piping mechanism.

Here's a sketch solution:

def zip() = Action {
  // Create Source that listens to an OutputStream
  // and pass it to `fileByteData` method.
  val zipSource: Source[ByteString, Unit] =
    StreamConverters
      .asOutputStream()
      .mapMaterializedValue(fileByteData)
  Ok.chunked(zipSource).withHeaders(
    CONTENT_TYPE -> "application/zip",
    CONTENT_DISPOSITION -> s"attachment; filename = test.zip")
}

// Send the file data, given an OutputStream to write to.
def fileByteData(os: OutputStream): Unit = {
  val fileList = List(
    new java.io.File("C:\\Users\\a.csv"),
    new java.io.File("C:\\Users\\b.csv")
  )

  val zos = new ZipOutputStream(os)
  val buffer: Array[Byte] = new Array[Byte](2048)
  try {
    for (file <- fileList) {
      zos.putNextEntry(new ZipEntry(file.toPath.getFileName.toString))
      val fis = new Files.newInputStream(file.toPath)
      try {
        @tailrec
        def zipFile(): Unit = {
          val bytesRead = fis.read(buffer)
          if (bytesRead == -1) () else {
            zos.write(buffer, 0, bytesRead)
            zipFile()
          }
        }
        zipFile()
      } finally fis.close()
      zos.closeEntry()
    }
  } finally {
    zos.close()
  }
}

This is just an outline of an approach. You'll also want to make sure: - the threading is OK - the fileByteData will hopefully run on a different thread to the sending thread - the error handling is OK - e.g. all streams are closed properly if there's an error on either the server (e.g. file not found) or client side (early disconnect)