Transforming JSON with Circe

Question

Suppose I have some JSON data like this:

{
    "data": {
        "title": "example input",
        "someBoolean": false,
        "innerData":  {
            "innerString": "input inner string",
            "innerBoolean": true,
            "innerCollection": [1,2,3,4,5]
        },
        "collection": [6,7,8,9,0]
    }
}

And I want to flatten it a bit and transform or remove some fields, to get the following result:

{
    "data": {
        "ttl": "example input",
        "bool": false,
        "collection": [6,7,8,9,0],
        "innerCollection": [1,2,3,4,5]
    }
}

How can I do this with Circe?

(Note that I'm asking this as a FAQ since similar questions often come up in the Circe Gitter channel. This specific example is from a question asked there yesterday.)

Answer 1

I've sometimes said that Circe is primarily a library for encoding and decoding JSON, not for transforming JSON values, and in general I'd recommend mapping to Scala types and then defining relationships between those (as Andriy Plokhotnyuk suggests here), but for many cases writing transformations with cursors works just fine, and in my view this kind of thing is one of them.

Here's how I'd implement this transformation:

import io.circe.{DecodingFailure, Json, JsonObject}
import io.circe.syntax._

def transform(in: Json): Either[DecodingFailure, Json] = {
  val someBoolean = in.hcursor.downField("data").downField("someBoolean")
  val innerData = someBoolean.delete.downField("innerData")

  for {
    boolean    <- someBoolean.as[Json]
    collection <- innerData.get[Json]("innerCollection")
    obj        <- innerData.delete.up.as[JsonObject]
  } yield Json.fromJsonObject(
    obj.add("boolean", boolean).add("collection", collection)
  )
}

And then:

val Right(json) = io.circe.jawn.parse(
  """{
    "data": {
      "title": "example input",
      "someBoolean": false,
      "innerData":  {
        "innerString": "input inner string",
        "innerBoolean": true,
        "innerCollection": [1,2,3]
      },
      "collection": [6,7,8]
    }
  }"""
)

And:

scala> transform(json)
res1: Either[io.circe.DecodingFailure,io.circe.Json] =
Right({
  "data" : {
    "title" : "example input",
    "collection" : [
      6,
      7,
      8
    ]
  },
  "boolean" : false,
  "collection" : [
    1,
    2,
    3
  ]
})

If you look at it the right way, our transform method kind of resembles a decoder, and we can actually write it as one (although I'd definitely recommend not making it implicit):

import io.circe.{Decoder, Json, JsonObject}
import io.circe.syntax._

val transformData: Decoder[Json] = { c =>
  val someBoolean = c.downField("data").downField("someBoolean")
  val innerData = someBoolean.delete.downField("innerData")

  (
    innerData.delete.up.as[JsonObject],
    someBoolean.as[Json],
    innerData.get[Json]("innerCollection")
  ).mapN(_.add("boolean", _).add("collection", _)).map(Json.fromJsonObject)
}

This can be convenient in some situations where you want to perform the transformation as part of a pipeline that expects a decoder:

scala> io.circe.jawn.decode(myJsonString)(transformData)
res2: Either[io.circe.Error,io.circe.Json] =
Right({
  "data" : {
    "title" : "example input",
    "collection" : [ ...

This is also potentially confusing, though, and I've thought about adding some kind of Transformation type to Circe that would encapsulate transformations like this without questionably repurposing the Decoder type class.

One nice thing about both the transform method and this decoder is that if the input data doesn't have the expected shape, the resulting error will include a history that points to the problem.