Search code examples
scalaapache-sparktypesafehocon

Load typesafe configs in HOCON format from custom locations at runtime

I have an application that will get job configurations informed thru files (likely to have dozens of them each describing a particular job to be done by the app, preferably being flexible about where they are located), and for that we chose HOCON format and typesafe libs.

It works well in development loading from the resources folder, but the intent is to pass the path to these configuration files at runtime thru a parameter while calling spark-submit (spark-submit ... pathToFile ...).

But reading the path to the file fails with this error:

val jConfig = ConfigFactory.load(path)

com.typesafe.config.ConfigException$Missing: system properties: No configuration setting found for key 'configuration'

How can I read config files from other locations than /resources?

Solution

As per @hagarwal tip, the following worked

val input = Source.fromFile(path,"UTF8").mkString
val jConfig = ConfigFactory.parseString(input)
Solution

  • Store config in HDFS/S3/ADFS/Local file system, pass Configuration file path to the program as argument, read config file as input stream (or file) form the configuration file path.

    val confPath = args(0) //Configuration file path
val stream = Client.getObject(confPath) //Client -> HDFS/S3/ADFS/Local
val configString = Source.fromInputStream(stream.asInstanceOf[InputStream]).mkString
val config = ConfigFactory.parseString(configString)