Recursive HTTP-requests in Scala

Question

I need to do recursive requests and then collect all models into one List, but not understand how to do it. Please tell me am I thinking right way?

package kindSir.main

import dispatch.Defaults._
import dispatch._
import kindSir.models._
import org.json4s._
import org.json4s.jackson.JsonMethods._

object ApplicationMain extends App {

  def fetchMergeRequests(startPage: Int = 1): Future[List[MergeRequest]] = {
    val requestsUrl = url(s"https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-ce/merge_requests?state=opened&per_page=3&page=${startPage}")
    Http(requestsUrl).map { res =>
      (parse(res.getResponseBody), res.getHeader("X-Next-Page").toInt) match {
        case (list@JArray(_), nextPage: Int) =>
          val currentList: List[MergeRequest] = MergeRequest.parseList(list).get
          val nextPageListFuture: Future[List[MergeRequest]] = fetchMergeRequests(nextPage)
          // And how to merge these two lists?
        case (list@JArray(_), _) => MergeRequest.parseList(list).get
        case _ => throw new RuntimeException(s"No merge requests for project found")
      }
    }
  }

}

Answer 1

The main problem you're dealing with here is that you're trying to combine data you already have (List[MergeRequest]) with the data you'll retrieve in future (Future[List[MergeRequest]]). There are a few things you need to do to handle this scenario:

• Use flatMap instead of map on result of the HTTP request. This allows you to make further HTTP requests inside the recursion but map them back to a single Future.
• Call map on the result of the recursion fetchMergeRequests(nextPage) to combine the data you already have with the future data from the recursion.
• Wrap the other list in Future.successful() because flatMap requires all the pattern matches to return a Future — except for the exception.

I'm not familiar with the libraries you are using so I haven't tested it, but I think your code should work like this:

def fetchMergeRequests(startPage: Int = 1): Future[List[MergeRequest]] = {
  val requestsUrl = url(s"https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-ce/merge_requests?state=opened&per_page=3&page=${startPage}")
  Http(requestsUrl).flatMap { res =>
    (parse(res.getResponseBody), res.getHeader("X-Next-Page").toInt) match {
      case (list@JArray(_), nextPage: Int) =>
        val currentList: List[MergeRequest] = MergeRequest.parseList(list).get
        val nextPageListFuture: Future[List[MergeRequest]] = fetchMergeRequests(nextPage)
        nextPageListFuture.map(nextPageList => currentList ++ nextPageList)
      case (list@JArray(_), _) =>
        Future.successful(MergeRequest.parseList(list).get)
      case _ => throw new RuntimeException(s"No merge requests for project found")
    }
  }
}