I have an complex object which contains another object. I want to write it as yaml file.I am using scala with ObjectMapper . My final object looks like this
Configuration(Some(List(s3://etl/configuration/minioApp/23-08-2021/metrics.yaml)),Some(Map(inputDataFrame -> Input(None,Some(Kafka(List(uguyhi, ytvvt),Some(hgvgvugb),Some(ytfytvi),Some(yftug),None,None,None,Some( uyguytf)))))),None,Some(Output(None,Some(Kafka(null,Some(yrdryft),None)))),None,None,None,None,None,None,None,None,Some(minioApp),None,None,None,None)
I want to write it to file. My main case class looks like this
case class Configuration(@BeanProperty var metrics: Option[Seq[String]],
@BeanProperty var inputs: Option[Map[String, Input]],
@BeanProperty var variables: Option[Map[String, String]] = None,
@BeanProperty var output: Option[Output] = None,
@BeanProperty var outputs: Option[Map[String, Output]] = None,
@BeanProperty var cacheOnPreview: Option[Boolean] = None,
@BeanProperty var showQuery: Option[Boolean] = None,
@BeanProperty var streaming: Option[Streaming] = None,
@BeanProperty var periodic: Option[Periodic] = None,
@BeanProperty var logLevel: Option[String] = None,
@BeanProperty var showPreviewLines: Option[Int] = None,
@BeanProperty var explain: Option[Boolean] = None,
@BeanProperty var appName: Option[String] = None,
@BeanProperty var continueOnFailedStep: Option[Boolean] = None,
@BeanProperty var cacheCountOnOutput: Option[Boolean] = None,
@BeanProperty var ignoreDeequValidations: Option[Boolean] = None,
@BeanProperty var failedDFLocationPrefix: Option[String] = None) extends Conf
and further input, output all class also have BeanProperty added. When I am trying to write this file using below snipplet.
val objectMapper = new ObjectMapper(new YAMLFactory)
objectMapper.writeValue(new File("/Users/ayush.goyal/Downloads/servingWrapper/src/main/resources/input1.yaml"),jobYamls)
where jobYaml is my object as mentioned above. I am getting below file.
metrics:
empty: false
defined: true
inputs:
empty: false
defined: true
variables:
empty: true
defined: false
output:
empty: false
defined: true
outputs:
empty: true
defined: false
cacheOnPreview:
empty: true
defined: false
showQuery:
empty: true
defined: false
streaming:
empty: true
defined: false
periodic:
empty: true
defined: false
logLevel:
empty: true
defined: false
showPreviewLines:
empty: true
defined: false
explain:
empty: true
defined: false
appName:
empty: false
defined: true
continueOnFailedStep:
empty: true
defined: false
cacheCountOnOutput:
empty: true
defined: false
ignoreDeequValidations:
empty: true
defined: false
failedDFLocationPrefix:
empty: true
defined: false
Now this file has 2 issue
How can I do that?
EDIT:
As per the suggestion, I was abl to get below yaml
---
metrics:
- "s3://etl/configuration/minioApp/23-08-2021/metrics.yaml"
inputs:
inputDataFrame:
file: null
kafka:
servers:
- "uguyhi"
- "ytvvt"
topic: "hgvgvugb"
topicPattern: "ytfytvi"
consumerGroup: "yftug"
options: null
schemaRegistryUrl: "yrftg"
schemaSubject: null
schemaId: " uyguytf"
output:
file: null
kafka:
vservers: null
checkpointLocation: "yrdryft"
compressionType: null
streaming: null
appName: "minioApp"
As you can see in above yaml, Several fields have null value. I don't want to write them. How can I do that? I just don't want those fiels to appear in Yaml.
Jackson doesn't know how to serialize Option as it's a Java library.
You can add the jackson-scala-module library and register the DefaultScalaModule on your ObjectMapper to let it know how to serialize common Scala types.