I am using lightbend in my scala code so that my application.properties file is away from my jar. I need to pass the current date in application.properties file.
Date_read_write = 20190828
how do I make this to pick up dynamically, if it is shell script I can mention
Date_read_write=`date +%Y%m%d`
how do I do a similar step in application.properties file?
Update for time addition
below is my application.properties file:
hdfs_link = hdfs://XX.XXX.XXX.XX:8080/sessions/data/events
mediaKey = 1234
eventName = media
Date_read_write = 20190815
time = *_W
parts = *
i am using above to generate hdfs://XX.XXX.XXX.XX:8080/sessions/data/events/1234/media/20190815/*_W/*
using :
val conf = ConfigFactory.load()
val hdfs_path = conf.getString("hdfs_link")+"/"+conf.getString("mediaKey")+"/"+conf.getString("eventName")+"/"+conf.getString("Date_read_write")+"/"+conf.getString("time")+"/"+conf.getString("parts")
but when i add
--driver-java-options -DDate_read_write=`date --date='1 days ago' '+%Y%m%d'`
to my spark submit commad, my URL is turing to hdfs://XX.XXX.XXX.XX:8080/sessions/data/events/1234/media/20190828/*/*
i have no clue why this addition changes the value of time.
Below is my spark submit command:
nohup spark-submit --master yarn --deploy-mode client --num-executors 20 --executor-cores 6 --driver-memory 10g --executor-memory 10g --class metrics.MasterAggregateTable --files application.properties --driver-java-options -Dconfig.file=application.properties --driver-java-options -DDate_read_write=`date +%Y%m%d` --jars com.datastax.spark_spark-cassandra-connector_2.11-2.3.0.jar UserMetrics.jar &
You can override all the settings in application.properties by system properties.
java -DDate_read_write=`date +%Y%m%d` -jar yourapp.jar
In default uses of the library, exact-match system properties already override the corresponding config properties.
This does not require any changes to the configuration file inside of the jar.