TensorBoard showing lots of 'nodes' from previous models

Question

I am training a model on the MNIST data and I am using tensorboard to visualise the training and validation loss.

Here is the code for my current model I am trying:

model=tf.keras.models.Sequential()
#callback=tf.keras.callbacks.EarlyStopping(monitor='accuracy', min_delta=0, patience=0, verbose=0, mode='auto',restore_best_weights=False)
#model.add(tf.keras.layers.InputLayer(input_shape=[28,28]))

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

reduce_lr=tf.keras.callbacks.ReduceLROnPlateau(
    monitor='val_loss', factor=0.1, patience=5, verbose=0, mode='auto',
    min_delta=0.0001, cooldown=0, min_lr=0)

optimizer=tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=True,
    name='Adam', clipnorm=5)

# if hparams[HP_OPTIMIZER]=='adam':
#     optimizer=tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=True,
#     name='Adam', clipnorm=5)

# elif hparams[HP_OPTIMIZER]=='sgd':
#     tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=False, name='SGD', **kwargs)
    

model.add(tf.keras.layers.Flatten(input_shape=[28,28]))
l2_new=tf.keras.regularizers.L2(
    l2=0.05)


model.add(tf.keras.layers.BatchNormalization(
    axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True,
    beta_initializer='zeros', gamma_initializer='ones',
    moving_mean_initializer='zeros', moving_variance_initializer='ones',
    beta_regularizer=None, gamma_regularizer=None, beta_constraint=None,
    gamma_constraint=None, renorm=True, renorm_clipping=None, renorm_momentum=0.99))



model.add(tf.keras.layers.Dense(300,activation='relu',kernel_initializer="he_normal",
        kernel_regularizer=l2_new, bias_regularizer=l2_new))

model.add(tf.keras.layers.BatchNormalization(
    axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True,
    beta_initializer='zeros', gamma_initializer='ones',
    moving_mean_initializer='zeros', moving_variance_initializer='ones',
    beta_regularizer=None, gamma_regularizer=None, beta_constraint=None,
    gamma_constraint=None, renorm=True, renorm_clipping=None, renorm_momentum=0.99))


model.add(tf.keras.layers.Dense(300,activation='relu',kernel_initializer="he_normal",
        kernel_regularizer=l2_new, bias_regularizer=l2_new))

model.add(tf.keras.layers.BatchNormalization(
    axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True,
    beta_initializer='zeros', gamma_initializer='ones',
    moving_mean_initializer='zeros', moving_variance_initializer='ones',
    beta_regularizer=None, gamma_regularizer=None, beta_constraint=None,
    gamma_constraint=None, renorm=True, renorm_clipping=None, renorm_momentum=0.99))


model.add(tf.keras.layers.Dense(10,activation='softmax'))

Sorry if it is a bit messy. I am essentially creating a sequential model with

1. A Flatten input layer
2. A Batch Norm layer 3.A 300 neuron dense layer
3. A Batch Norm layer
4. A 300 neuron dense layer
5. A Batch Norm layer
6. A Softmax output layer with 10 neurons.

My model also uses the 'Adam' optimizer and learning rate decay.

When I view my model under the graphs sub heading in tensorboard, I get the following picture:

As you can see, there are lots of 'nodes', which I am guessing is because I have trained multiple models. How do I get rid of all the previous attempts.

I have tried using del model and tf.keras.backend.clear_session() but they didn't work.

Edit: I have followed the advice of 'Aniket Bote' and deleted the logs. Here is the new output:

I am still not sure it is correct. From my code, I don't think that my graph should have 2 branches as is denoted, and I am still getting that huge stack of batch normalisation 'nodes' to the right.

Answer 1

The second branch is not a graph in itself but rather it is a subgraph.
Tensorflow build graphs of the operation it performs in order to speed up the execution of code. If you click on those you can see they are functions that are utilized by the batch normalization layer, not the layer itself. You can see all the layer information on your main graph.

If you don't want those nodes you can get rid of them by setting the BatchNormalization's trainable attribute as False.

In this case, the layer's weight won't change and TensorFlow would no longer require to compute anything for that layer ie. No function nodes will be generated.

Code:

import tensorflow as tf
import numpy as np

np.random.seed(100)
x = tf.constant(np.random.randint(50, size =(1000,28,28)), dtype = tf.float32)
y = tf.constant(np.random.randint(10, size =(1000,)), dtype = tf.int32)

model=tf.keras.models.Sequential()
log_dir = "logs"
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1, profile_batch = 0)

optimizer=tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=True,
    name='Adam', clipnorm=5)

model.add(tf.keras.layers.Flatten(input_shape=[28,28]))
l2_new=tf.keras.regularizers.L2(
    l2=0.05)


model.add(tf.keras.layers.BatchNormalization(
    axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True,
    beta_initializer='zeros', gamma_initializer='ones',
    moving_mean_initializer='zeros', moving_variance_initializer='ones',
    beta_regularizer=None, gamma_regularizer=None, beta_constraint=None,
    gamma_constraint=None, renorm=True, renorm_clipping=None, renorm_momentum=0.99,trainable = False))



model.add(tf.keras.layers.Dense(300,activation='relu',kernel_initializer="he_normal",
        kernel_regularizer=l2_new, bias_regularizer=l2_new))

model.add(tf.keras.layers.BatchNormalization(
    axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True,
    beta_initializer='zeros', gamma_initializer='ones',
    moving_mean_initializer='zeros', moving_variance_initializer='ones',
    beta_regularizer=None, gamma_regularizer=None, beta_constraint=None,
    gamma_constraint=None, renorm=True, renorm_clipping=None, renorm_momentum=0.99,trainable = False))


model.add(tf.keras.layers.Dense(300,activation='relu',kernel_initializer="he_normal",
        kernel_regularizer=l2_new, bias_regularizer=l2_new))

model.add(tf.keras.layers.BatchNormalization(
    axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True,
    beta_initializer='zeros', gamma_initializer='ones',
    moving_mean_initializer='zeros', moving_variance_initializer='ones',
    beta_regularizer=None, gamma_regularizer=None, beta_constraint=None,
    gamma_constraint=None, renorm=True, renorm_clipping=None, renorm_momentum=0.99,trainable = False))


model.add(tf.keras.layers.Dense(10,activation='softmax'))


model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
tf.keras.utils.plot_model(model, 'my_first_model.png', show_shapes=True)

model.fit(x,y, epochs = 10, callbacks = tensorboard_callback)

Output:

Epoch 1/10
32/32 [==============================] - 0s 10ms/step - loss: 89.0275 - accuracy: 0.1100
Epoch 2/10
32/32 [==============================] - 0s 9ms/step - loss: 56.7906 - accuracy: 0.1310
Epoch 3/10
32/32 [==============================] - 0s 9ms/step - loss: 48.5681 - accuracy: 0.1490
Epoch 4/10
32/32 [==============================] - 0s 9ms/step - loss: 42.8176 - accuracy: 0.1850
Epoch 5/10
32/32 [==============================] - 0s 9ms/step - loss: 38.5857 - accuracy: 0.2110
Epoch 6/10
32/32 [==============================] - 0s 9ms/step - loss: 35.1675 - accuracy: 0.2540
Epoch 7/10
32/32 [==============================] - 0s 9ms/step - loss: 32.3327 - accuracy: 0.2750
Epoch 8/10
32/32 [==============================] - 0s 9ms/step - loss: 29.8839 - accuracy: 0.3420
Epoch 9/10
32/32 [==============================] - 0s 9ms/step - loss: 27.7426 - accuracy: 0.3940
Epoch 10/10
32/32 [==============================] - 0s 10ms/step - loss: 25.6565 - accuracy: 0.4930

Tensorboard Graph Image: