Building a dense residual network with keras

Question

I am trying to build a classifier based on a Dense Network with Keras. My input are (26,1) vectors and I want to get a binary classification 1 or 0 as output.

Using a Dense network and some optimisation with hyperas I manage to reach 80% accuracy which is not bad but I am trying to improve the accuracy of the network using Residual networks.

I found on various forums a lot of examples of residual networks for convolutionary networks but I did not find examples of residual networks.

I tried the following code to generate the residual net :

Input=keras.Input(shape=(X_train.shape[1],))
x1=layers.Dense(512,activation='relu')(Input)
x1=layers.Dropout(0.5)(x1)
x1=layers.Dense(128,activation='relu')(x1)
x1=layers.Dropout(0.4)(x1)
# x1=layers.Dense(512,activation='relu')(x1)
# x1=layers.Dropout(0.3)(x1)

x2=layers.Dense(128,activation='relu')(Input)
x2=layers.Dropout(0.5)(x2)

added=layers.Add()([x1,x2])#pour faire du ResNet
x3=layers.Dense(128,activation='relu')(added)
final=layers.Dense(1,activation='sigmoid')(x3)

model=Model(Input,final)

model.compile(optimizer='RMSprop',loss='binary_crossentropy',metrics=['accuracy'])

history=model.fit(X_train,Y_train,validation_data=(X_valid,Y_valid),epochs=100,batch_size=128,class_weight=class_weight)

loss = history.history['acc']
val_loss=history.history['val_acc']

epochs=range(1,len(loss)+1)

plt.plot(epochs,loss,'bo',label='Training accuracy')
plt.plot(epochs,val_loss,'b',label='Validation accuracy')
plt.title('Training and validation accuracy')
plt.legend()

plt.show()

I tried various epoch training but the net does not reach accuracy over 75% which is worse than before. Of course I can still use hyperas to improve again accurcy and adjust hyperparameters but I was expecting somewhat better performances "at first".

Questions :

• is there a flaw in my coding ?
• Are there better ways to generate the residual net ? Mainly, I added one skip layer (which still passes through one dense layer) could i do better ? Should I include more ?

Thanks for your advices

Answer 1

Firstly, resnets are intended to be used in much deep networks. Your network seems too shallow to get maximum adding benefit.

More generally, resnets are kind of an evolved version of simple architectures like VGGNet, with the intend to be able to "go deeper". That does not mean that residual layers will always increase your accuracy, if the network is too shallow.

Adding more layers should help. The key idea is to avoid vanishing gradients in the deeper layers of the network by allowing to skip connections.