How to read all the properties files with given prefix in spark, spring application?

Question

I am developing an application in spark and scala and using spring to read configuration files.

My Environment specific files available in that directory like this

src/main/resource/DEV

      mms_kafka.properties

      mms_app.properties

      pps_kafka.properties

      pps_app.properties

And common files under src/main/resoruce like below

src/main/resource

mmsmappings.properties 

ppsmappings.properties

Currently, I am doing like below and working fine

@PropertySource(value = Array("classpath:${ENV}/mms_app.properties","classpath:${ENV}/mms_kafka.properties","classpath:$mmsmapping.properties"), ignoreResourceNotFound=false)

Spark submit command: spark2-submit --master yarn --deploy-mode client --class job.Driver --conf 'spark.driver.extraJavaOptions=-DENV=DEV' --driver-memory 4g --executor-memory 16g --num-executors 4 --executor-cores 4  temp-0.0.1-shaded.jar 

But I want to read all the files for a particular prefix(mms/pps) like below , I tried it but it is giving ENV and APP place holder is not resolve

@PropertySource(value = Array("classpath:${ENV}/${APP}_app.properties","classpath:${ENV}/${APP}_kafka.properties","classpath:${APP}mapping.properties"), ignoreResourceNotFound=false)


Spark submit command: spark2-submit --master yarn --deploy-mode client --class job.Driver --conf 'spark.driver.extraJavaOptions=-DENV=DEV' --conf 'spark.driver.extraJavaOptions=-DAPP=mms' --driver-memory 4g --executor-memory 16g --num-executors 4 --executor-cores 4  temp-0.0.1-shaded.jar

How should I fix this?

Answer 1

I solved this by passing parameters in spark job like below

spark2-submit --master yarn --deploy-mode client --class com.job.Driver --conf 'spark.driver.extraJavaOptions=-DENV=DEV -DAPP=mms' --driver-memory 4g --executor-memory 16g --num-executors 4  --executor-cores 4 test.jar