Search code examples
neural-networkdeep-learningtflearn

Decrease total-loss in Deep neural network

I use tflearn.DNN to build a deep neural network:

# Build neural network
net = tflearn.input_data(shape=[None, 5], name='input')
net = tflearn.fully_connected(net, 64, activation='sigmoid')
tflearn.batch_normalization(net)
net = tflearn.fully_connected(net, 32, activation='sigmoid')
tflearn.batch_normalization(net)
net = tflearn.fully_connected(net, 16, activation='sigmoid')
tflearn.batch_normalization(net)
net = tflearn.fully_connected(net, 8, activation='sigmoid')
tflearn.batch_normalization(net)
# activation needs to be softmax for classification.
# default loss is cross-entropy and the default metric is accuracy
# cross-entropy + accuracy = categorical network
net = tflearn.fully_connected(net, 2, activation='softmax')
sgd = tflearn.optimizers.SGD(learning_rate=0.01, lr_decay=0.96, decay_step=100)
net = tflearn.regression(net, optimizer=sgd, loss='categorical_crossentropy')

model = tflearn.DNN(net, tensorboard_verbose=0)

I tried many things, but all the time the total loss is around this value:

Training Step: 95  | total loss: 0.68445 | time: 1.436s
| SGD | epoch: 001 | loss: 0.68445 - acc: 0.5670 | val_loss: 0.68363 - val_acc: 0.5714 -- iter: 9415/9415

What can I do to decrease the total loss and make the accuracy get higher?

Solution

  • Many aspects can be considered to improve the network performance, including the datasets and the network. Just by the network structure you pasted, it is difficult to give a clear way to increase its accuracy without more info about datasets and the target you want to get. But the following are some useful practices may help you to debug / improve the network:

    1. About the datasets

    • Is the datasets balanced with distortions?
    • Get more training data .
    • Add data augmentation if possible.
    • Normalising data.
    • Feature engineering.

    2. About the network

    • Is the network size is too small / large?
    • Check overfitting or underfitting by train history, then chose the best epoch size.
    • Try initialise weights with different initialization scheme.
    • Try different activation functions, loss function, optimizer.
    • Change layers number and units number.
    • Change batch size.
    • Add dropout layer.

    And for more deeply analyse, the following articles may be helpful to you: