I tried implementing a char-LSTM classification model on a dummy dataset using CNTK and TF and both models achieve 100% accuracy on a perfect dataset (I am not concerned about overfitting for now).
However, once the training is done and the models are saved to a file and restored, the models seem to forget their training and perform poorly on the same dataset.
I have compared the weights and biases of the LSTMs and the dense layer after training and after restoring and they match perfectly. I feel there is something else that needs to be restored (LSTM states maybe?) for the models to start functioning accurately again.
Do restored LSTM models need to be "primed" in some way before they can start performing at full capacity again?
Code and dataset is available here.
The issue is not related to saving/loading LSTM.
In Python, when you convert a set to a list, the result is not sorted and different from run to run:
In [1]: list(set(['t', 'h', 'i', 's', ' ', 'i', 's', ' ', 'a', ' ', 's', 'p', 'a', 'm']))
Out[1]: ['s', 'a', 'm', 't', 'h', 'p', 'i', ' ']
In [1]: list(set(['t', 'h', 'i', 's', ' ', 'i', 's', ' ', 'a', ' ', 's', 'p', 'a', 'm']))
Out[1]: [' ', 'h', 'a', 't', 'm', 'i', 'p', 's']
Because of this, in your code different characters get different positions in the vector from run to run, so the performance is just random.
Simply changing all_chars = list(set(all_chars)) to all_chars = sorted(set(all_chars)) fixes the issue (I only verified for the CNTK script).