Search code examples
scalaakkaakka-streamakka-http

Scala read large set of chunk data through akka streaming


I am trying to read large chunk of data through scala as explained here : Scala read continuous http stream

import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.{Uri, HttpRequest}
import akka.stream.ActorMaterializer

object ChunkTestClient extends App {

  implicit val system = ActorSystem("test")
  import system.dispatcher

  implicit val materializer = ActorMaterializer()
  val source = Uri("https://jigsaw.w3.org/HTTP/ChunkedScript")
  val finished = Http().singleRequest(HttpRequest(uri = source)).flatMap { response =>
    response.entity.dataBytes.runForeach { chunk =>
      println(chunk.utf8String)
    }
  }
}

The above code snippet works fine for most of case. But in case the response data is huge, or source is continuously streaming large set of data, then chunk response does not work, neither it throw any error, it just kills silently. This seems to be case of back-pressure where rate of data production is grater than rate of consumption. Is there any way to handle large size\streaming of data with akka http streaming. What can be done in above code to handle back pressure at client side.


Solution

  • I finally managed to work this withoutSizeLimit(). I also tried withSizeLimit() but that does not worked for me.

    `

    object ChunkTestClient extends App {
    
      implicit val system = ActorSystem("test")
      import system.dispatcher
    
      implicit val materializer = ActorMaterializer()
      val source = Uri("https://jigsaw.w3.org/HTTP/ChunkedScript")
      val finished = Http().singleRequest(HttpRequest(uri = source)).flatMap { response =>
        response.entity.withoutSizeLimit().dataBytes.runForeach { chunk =>
          println(chunk.utf8String)
        }
      }
    }
    

    `