Given the following minimal example:
import numpy as np
import tensorflow as tf
from sklearn.model_selection import train_test_split
from tensorflow.python.keras.layers import Dense
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.optimizer_v2.adam import Adam
negatives = np.random.beta(0.6, 1.2, 100)
positives = np.random.beta(1.7, 0.9, 100)
x = np.concatenate((negatives, positives))
y = np.concatenate((np.full(negatives.size, 0), np.full(positives.size, 1)))
model = Sequential()
model.add(Dense(1, input_shape=(1,), activation="sigmoid"))
model.compile(optimizer=Adam(learning_rate=0.1),
loss="binary_crossentropy",
metrics=[tf.keras.metrics.AUC(curve="ROC", name="AUROC")])
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=42)
model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=100, verbose=True, batch_size=10000))
auroc_val = tf.keras.metrics.AUC(curve="ROC", name="AUROC")(y_test, model.predict(x_test).flatten()).numpy()
print(f"{auroc_val=}")
I'd expect auroc_val to be equal to val_AUROC (from the last training epoch). However, in reality, it is not.
Output of one run:
...
Epoch 98/100
5/5 [==============================] - 0s 5ms/step - loss: 0.5247 - AUROC: 0.8144 - val_loss: 0.5277 - val_AUROC: 0.8143
Epoch 99/100
5/5 [==============================] - 0s 5ms/step - loss: 0.5242 - AUROC: 0.8143 - val_loss: 0.5290 - val_AUROC: 0.8144
Epoch 100/100
5/5 [==============================] - 0s 5ms/step - loss: 0.5250 - AUROC: 0.8145 - val_loss: 0.5264 - val_AUROC: 0.8145
auroc_val=0.83203566
Output of another run:
...
Epoch 100/100
5/5 [==============================] - 0s 5ms/step - loss: 0.4771 - AUROC: 0.8055 - val_loss: 0.6642 - val_AUROC: 0.8054
auroc_val=0.7224694
Why is this?
I could not replicate your error, for me the values are the same. But I had to delete the from tensorflow.python.. imports and create e.g. the Dense layer with tf.keras.layers.Dense. See this explanation for details on why not to use tensorflow.python.
My TF version is 2.13.0 (which would explain the tf.python.keras import error), python 3.10.11. Please try without the imports mentioned above and check your TF version.
(This answer was originally only the comment under the question.)