I am trying to tune the hyperparameters of my fully connected deep learning model using flags and tuning_run in R using the keras package. Where do I find the actual flag value used in each run?
I have tried looking for the hyperparameter values used in both the generated result data frame and the runs/ folder. While all the accuracy values, loss function and other meta details about the runs are there, the hyperparameters for which those results are generated are not included (I followed this example given here: https://tensorflow.rstudio.com/tools/tfruns/articles/tuning.html). I am calling my tuning_run as given below
runs <- tuning_run("test.R", flags = list(dropout1=c(0.5,0.4,0.3),dropout2=c(0.3,0.2),dense_units=c(128,256)),sample=0.3)
and my model consumes the flags like
model <- keras_model_sequential()
model %>%
layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
layer_dropout(rate = FLAGS$dropout_1) %>%
layer_dense(units = FLAGS$dense_units, activation = 'relu') %>%
layer_dropout(rate = FLAGS$dropout_2) %>%
layer_dense(units = 10, activation = 'softmax')
When I run it, and later look for the value of the flags for which a certain validation accuracy is generated for (the runs dataframe) This is what I observe
Data frame: 2 x 25
run_dir eval_loss eval_acc metric_loss metric_acc
1 runs/2019-03-29T00-14-10Z 0.1315 0.9794 0.0075 0.9977
2 runs/2019-03-29T00-10-37Z 0.1326 0.9816 0.0096 0.9973
metric_val_loss metric_val_acc
1 0.1475 0.9794
2 0.1443 0.9794
# ... with 18 more columns:
# samples, validation_samples, batch_size, epochs, epochs_completed,
# metrics, model, loss_function, optimizer, learning_rate, script, start,
# end, completed, output, source_code, context, type
I am wondering where to find the flag values used in each iteration. Or am I doing something wrong? Any help would be appreciated. Thanks!
I found out what the problem was. The flags need to be defined in the target script too for keras to report it. And that was why it wasn't showing the flags in the resulting frame.
Once I added these lines to the test.R it worked fine
FLAGS <- flags(
flag_numeric('dropout_1', 0.04, 'First dropout'),
flag_numeric('dropout_2', 0.3, 'Second dropout'),
flag_integer('dense_units', 128, 'Units in dense layer')
)
The same problem and the solution is discussed here: https://github.com/rstudio/tfruns/issues/24