I train my model in multi steps. When I train a first round, change the learning rate and train again, all work fine.
but
When I train the first round, load the best epoch trained model and change the learning rate I got this error:
File "E:\Anaconda3\envs\tf2.7_bigData\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "E:\Anaconda3\envs\tf2.7_bigData\lib\site-packages\h5py\_hl\group.py", line 161, in create_dataset
dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
File "E:\Anaconda3\envs\tf2.7_bigData\lib\site-packages\h5py\_hl\dataset.py", line 156, in make_new_dset
dset_id = h5d.create(parent.id, name, tid, sid, dcpl=dcpl, dapl=dapl)
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5d.pyx", line 87, in h5py.h5d.create
ValueError: Unable to create dataset (name already exists)
This is the way I load my model and change the learning rate:
model = tensorflow.keras.models.load_model('resultats/' + str(bv_num) + '_convergeance_' + nom_exp + '.h5')
optimizer.lr.assign(learning_rates[1])
model.compile(optimizer=optimizer, loss='mse')
Optimizer was created earlyer this way:
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
I try to remove learning rate change and the error don't disapear, so it seem to be loading the model who create the problem.
I seen a lot of discussion about similar error, and the need of naming each layer with uniq name (what a did) but with out loading the model all work great so I imagine this is not the trouble...
It seem is the way I compile my model. I re-compile the model later to change the learning rate. When removing this double compile step all run great.