First: I'm not able to provide any code - if that's enough reason to close this - so be it.
The codebase I was provided is confidential and I was not able to reproduce this behavior in a standalone example after multiple hours of work. It's too much custom code that I barely understand which makes isolating single parts basically impossible.
That's why I'm now trying the opposite approach: looking at the tensorflow code to find out what variable combination has to happen to cause this behaviour and then work from there.
I'm asking for advice on how to debug this issue myself / or if anyone has seen something like this:
The actual Issue:
I'm training a tensorflow model with model.fit and set the validation_freq argument to e.g. 10. The behavior I get is that validation is performed on epoch 10 and that's it. On epoch 20, 30 ... and so on no validation is performed. To summarise: for each model.fit call I get only one evaluation in total.
Tensorflow Version: 2.10.0
I tried going directly into site-packages/tensorflow/python/keras/engine/training.py to add print statements inside model.fit itself: here and here but I got no output so they were either not executed or print from this file doesn't reach stdout for some reason... or... I misunderstood the code flow of TensorFlow ...
I was changing site-packages/tensorflow/python/keras/engine/training.py while the code was actually using site-packages/keras/engine/training.py
What could cause this / how could I debug this?
It's hard to tell exactly the issue if you're unable to provide examples but I would try the following.
fit methodmodel.fit(...validation_freq=[10, 20, 30])