I am quite new to deep learning and I was studying this RNN example.
After completing the tutorial, I decided to see the effect of various hyperparameters such as the number of nodes in each layer and dropout factor etc.
What I do is, for each value in my lists, create a new model using a set of parameters and test the performance in my dataset. Below is the basic code:
def build_model(MODELNAME, l1,l2,l3, l4, d):
tf.global_variables_initializer()
tf.reset_default_graph()
model = Sequential(name = MODELNAME)
model.reset_states
model.add(CuDNNLSTM(l1, input_shape=(x_train.shape[1:]), return_sequences=True) )
model.add(Dropout(d))
model.add(BatchNormalization())
model.add(CuDNNLSTM(l2, input_shape=(x_train.shape[1:]), return_sequences=True) )
# Definition of other layers of the model ...
model.compile(loss="sparse_categorical_crossentropy",
optimizer=opt,
metrics=['accuracy'])
history = model.fit(x_train, y_train,
epochs=EPOCHS,
batch_size=BATCH_SIZE,
validation_data=(x_validation, y_validation))
return model
layer1 = [64, 128, 256]
layer2,3,4 = [...]
drop = [0.2, 0.3, 0.4]
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
for l1 in layer1:
#for l2, l3, l4 for layer2, layer3, layer4
for d in drop:
sess = tf.Session(config=config)
set_session(sess)
MODELNAME = 'RNN-l1={}-l2={}-l3={}-l4={}-drop={} '.format(l1, l2, l3, l4, d)
print(MODELNAME)
model = build_model(MODELNAME, l1,l2,l3, l4, d)
sess.close()
print('-----> training & validation loss & accuracies)
The problem is when the new model is built using the new parameters, it works as if the next epoch of the previous model, rather than epoch 1 of the new one. Below is some of the results.
RNN-l1=64-l2=64-l3=64-l4=32-drop=0.2
Train on 90116 samples, validate on 4458 samples
Epoch 1/6
90116/90116 [==============================] - 139s 2ms/step - loss: 0.5558 - acc: 0.7116 - val_loss: 0.8857 - val_acc: 0.5213
... # results for other epochs
Epoch 6/6
RNN-l1=64-l2=64-l3=64-l4=32-drop=0.3
90116/90116 [==============================] - 140s 2ms/step - loss: 0.5233 - acc: 0.7369 - val_loss: 0.9760 - val_acc: 0.5336
Epoch 1/6
90116/90116 [==============================] - 142s 2ms/step - loss: 0.5170 - acc: 0.7403 - val_loss: 0.9671 - val_acc: 0.5310
... # results for other epochs
90116/90116 [==============================] - 142s 2ms/step - loss: 0.4953 - acc: 0.7577 - val_loss: 0.9587 - val_acc: 0.5354
Epoch 6/6
90116/90116 [==============================] - 143s 2ms/step - loss: 0.4908 - acc: 0.7614 - val_loss: 1.0319 - val_acc: 0.5397
# -------------------AFTER 31TH SET OF PARAMETERS
RNN-l1=64-l2=256-l3=128-l4=32-drop=0.2
Epoch 1/6
90116/90116 [==============================] - 144s 2ms/step - loss: 0.1080 - acc: 0.9596 - val_loss: 1.8910 - val_acc: 0.5372
As seen, the first epoch of 31th set of parameters behaves as if it is 181th epoch. Similarly, if I stop the run at one point and re-run again, the accuracy and loss look as if it is the next epoch as below.
Epoch 1/6
90116/90116 [==============================] - 144s 2ms/step - loss: 0.1053 - acc: 0.9621 - val_loss: 1.9120 - val_acc: 0.5375
I tried a bunch of things (as you can see in the code), such as model=None, reinitializing the variables, resetting_status of the model, closing session in each iteration etc but none helped. I searched for similar question with no luck.
I am trying to understand what I am doing wrong. Any help is appreciated,
Note: Title is not very explanatory, I am open to suggestions for a better title.
Looks like you are using a Keras setting, which means you need to import keras backend and then clear that session before you run your new model. It would be something like this:
from keras import backend as K
K.clear_session()