Parsing a file having content as Json format in Scala

Question

I want to parse a file having content as json format. From the file I want to extract few properties (name, DataType, Nullable) to create some column names dynamically. I have gone through some examples but most of them are using case class but my problem is every time I will receive a file may have different content.

I tried to use the ujson library to parse the file but I am unable to understand how to use it properly.

object JsonTest {
  def main(args: Array[String]): Unit = {

    val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
    println(source)
    val input = try source.mkString finally source.close()
    println(input)

    val data = ujson.read(input)
    data("name") = data("name").str.reverse
    val updated = data.render()
  }
}

Content of the file example:

{
"Organization": {
"project": {
"name": "POC 4PL",
"description": "Implementation of orderbook"
},
"Entities": [
{
"name": "Shipments",
"Type": "Fact",
"Attributes": [
{
"name": "Shipment_Details",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "Shipment_ID",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "View_Cost",
"DataType": "StringType",
"Nullable": "true"
}
],
"ADLS_Location": "/mnt/mns/adls/raw/poc/orderbook/"
}
]
}
}

Expected output:

StructType(
Array(StructField("Shipment_Details",StringType,true),
StructField("Shipment_ID",DateType,true),   
StructField("View_Cost",DateType,true))) 

StructType needs to be added to the expected output programatically.

Answer 1

Try Using Playframework's Json utils - https://www.playframework.com/documentation/2.7.x/ScalaJson

Here's the solution to your issue- \ Placed your json in text file

    val fil_path = "C:\\TestData\\Config\\Conf.txt"
    val conf_source = scala.io.Source.fromFile(fil_path)
    lazy val json_str = try conf_source.mkString finally conf_source.close()
    val conf_json: JsValue = Json.parse(json_str)
    val all_entities: JsArray = (conf_json \ "Organization" \ "Entities").get.asInstanceOf[JsArray]
    val shipments: JsValue = all_entities.value.filter(e => e.\("name").as[String] == "Shipments").head
    val shipments_attributes: IndexedSeq[JsValue] = shipments.\("Attributes").get.asInstanceOf[JsArray].value
    val shipments_schema: StructType = StructType(shipments_attributes.map(a => Tuple3(a.\("name").as[String], a.\("DataType").as[String], a.\("Nullable").as[String]))
      .map(x => StructField(x._1, StrtoDatatype(x._2), x._3.toBoolean)))
    shipments_schema.fields.foreach(println)

Output is -

StructField(Shipment_Details,StringType,true)
StructField(Shipment_ID,StringType,true)
StructField(View_Cost,StringType,true)