How can I deserialize an non-fixed array of jsons using Circe's manual decoder?

Question

I have a JSON that looks like:

{
  "data": [
    {
      "id": "1",
      "email": "hello@world.com",
      "name": "Mr foo",
      "roles": [
        "Chief Bar Officer"
      ],
      "avatar_url": null,
      "phone_number": null
    },
    {
      "id": "2",
      "email": "bye@world.com",
      "name": "Mr baz",
      "roles": [
        "Chief Baz Officer"
      ],
      "avatar_url": null,
      "phone_number": null
    }
  ]
}

I am mainly interested in parsing/deserializing the data list, and I would like to do that manually (I prefer the manual way for some mysterious reason).

In case this is relevant, I am using sttp's circe library sttp.client.circe._ with the intention of parsing incoming data from get requests directly into Json using asJson.

A get sttp request looks something like:

val r1 = basicRequest
    .get(uri"https://woooo.woo.wo/v1/users")
    .header("accept", "application/json")
    .header("Authorization", "topsecret"
    .response(asJson[SomeClass])

This is what I have tried so far:

// Define the case class
case class User(
    id: String,
    email: String,
    name: String,
    roles: List[String],
    avatar_url: Option[String],
    phone_number: Option[String]
)

// Define the manual deserializer

case object User {

  implicit val userDecoder: Decoder[User] = (hCursor: HCursor) => {
    val data = hCursor.downField("data").downArray
    for {
      id <- data.get[String]("id")
      email <- data.get[String]("email")
      name <- data.get[String]("name")
      roles <- data.get[List[String]]("roles")
      avatarUrl <- data.get[Option[String]]("avatarUrl")
      phoneNumber <- data.get[Option[String]]("phoneNumber")
    } yield User(id, email, name, roles, avatarUrl, phoneNumber)
  }
}

The problem with my approach (I think) is that .downArray makes me only serialize the first User in the array of Users.

My objective is to be able to have some sequence of users (something like List[User] maybe), but at the moment I only end up deserializing one user in the array.

It is worth mentioning that the "data" array, does not contain a fixed-number of users, and every api call could result in a different number of users.

Answer 1

Thanks to the help of Travis Brown and the circe Gitter community for helping me figure this one out.

I'm quoting Travis here:

it would be better to build up the instance you need to parse the top-level JSON object compositionally… i.e. have a Decoder[User] that only decodes a single user JSON object, and then use Decoder[List[User]].at("data") or something similar to decode the top-level JSON object containing the data field with the JSON array.

I have ended up with an implementation that looks something like:

case class Users(users: List[User])

case object User {

  implicit val usrDecoder: Decoder[User] = (hCursor: HCursor) => {

    for {
      id <- hCursor.get[String]("id")
      email <- hCursor.get[String]("email")
      name <- hCursor.get[String]("name")
      roles <- hCursor.get[List[String]]("roles")
      avatarUrl <- hCursor.get[Option[String]]("avatarUrl")
      phoneNumber <- hCursor.get[Option[String]]("phoneNumber")
    } yield User(id, email, name, roles, avatarUrl, phoneNumber)
  }

  implicit val decodeUsers: Decoder[Users] =
    Decoder[List[User]].at("data").map(Users)

}

The idea is to compose the Decoder of a User, and the Decoder for a collection of Users separately. And then by mapping Users to the Decoder, we wrap the results of the Decoder into the Users case class.