How to add more layers to existing model (eg. teachable machine application model)?

Question

I'm trying to use the google model from teachable machine application https://teachablemachine.withgoogle.com/ by adding few more layers before output layers. When I retrain the model, always return this error:

ValueError: Input 0 of layer dense_25 is incompatible with the layer: expected axis -1 of input shape to have value 5 but received input with shape [20, 512]

Here's my approach:

When retrain the model it return error:

If I retrain the model without adding new layers, it's working fine. Can anybody advise what was the issue?

Answer 1

UPDATED ANSWER

if you want to add layers in between two layers for a pre-trained model, it is not as straightforward as adding layers using add method. if done so will result in un-expected behavior

analysis of error:

if you compile the model like below(like you specified):

model.layers[-1].add(Dense(512, activation ="relu"))
model.add(Dense(128, activation="relu"))
model.add(Dense(32))
model.add(Dense(5))

output of model summary :

Model: "sequential_12"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
sequential_9 (Sequential)    (None, 1280)              410208    
_________________________________________________________________
sequential_11 (Sequential)   (None, 512)               131672    
_________________________________________________________________
dense_12 (Dense)             (None, 128)               768       
_________________________________________________________________
dense_13 (Dense)             (None, 32)                4128      
_________________________________________________________________
dense_14 (Dense)             (None, 5)                 165       
=================================================================
Total params: 546,941
Trainable params: 532,861
Non-trainable params: 14,080
_________________________________________________________________

everything looks good here, but on closer look :

for l in model.layers:
  print("layer : ", l.name, ", expects input  of shape : ",l.input_shape)

output :

layer :  sequential_9 , expects input  of shape :  (None, 224, 224, 3)
layer :  sequential_11 , expects input  of shape :  (None, 1280)
layer :  dense_12 , expects input  of shape :  (None, 5) <-- **PROBLEM**
layer :  dense_13 , expects input  of shape :  (None, 128)
layer :  dense_14 , expects input  of shape :  (None, 32)

PROBLEM here is that dense_12 expects an input of shape(None, 5) but it should expect input shape of (None, 512) since we have added Dense(512) to sequential_11, possible reason would be adding layers like above specified might not update few attributes such as output shape of sequential_11, so during forward pass there is as miss-match between output of sequential_11 and input of layer dense_12(in your case dense_25)

possible work around would be :

for your question "adding layers in between sequential_9 and sequential_11", you can add as many layers as you want in between sequential_9 and sequential_11, but always make sure that output shape of last added layer should match input shape expected by sequential_11. in this case it is 1280.

code :

sequential_1 = model.layers[0] # re-using pre-trained model
sequential_2 = model.layers[1]

from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Model

inp_sequential_1 = Input(sequential_1.layers[0].input_shape[1:])
out_sequential_1 = sequential_1(inp_sequential_1)

#adding layers in between sequential_9 and sequential_11
out_intermediate = Dense(512, activation="relu")(out_sequential_1)
out_intermediate = Dense(128, activation ="relu")(out_intermediate)
out_intermediate = Dense(32, activation ="relu")(out_intermediate)

# always make sure to include a layer with output shape matching input shape of sequential 11, in this case 1280
out_intermediate = Dense(1280, activation ="relu")(out_intermediate)

output = sequential_2(out_intermediate) # output of intermediate layers are given to sequential_11 

final_model = Model(inputs=inp_sequential_1, outputs=output)

output of model summary:

Model: "functional_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_5 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
sequential_9 (Sequential)    (None, 1280)              410208    
_________________________________________________________________
dense_15 (Dense)             (None, 512)               655872    
_________________________________________________________________
dense_16 (Dense)             (None, 128)               65664     
_________________________________________________________________
dense_17 (Dense)             (None, 32)                4128      
_________________________________________________________________
dense_18 (Dense)             (None, 1280)              42240     
_________________________________________________________________
sequential_11 (Sequential)   (None, 5)                 128600    
=================================================================
Total params: 1,306,712
Trainable params: 1,292,632
Non-trainable params: 14,080