Search code examples
tensorflowdeep-learningloss-function

How to use tensorflow to build a deep neural network with the local loss for each layer?

I'm a novice on tensorflow (TF). Recently, I feel confused when I try to use TF to construct my deep model, in which each layer has its own (local) loss function.

It seems like many deep models (e.g., CNN) implemented by TF has only one (global) loss function, so one can first comput the hidden representations from the input layer to output layer; then obtain the loss; finally, using the existing optimation algorithm and loss to train model.

Unlike those models, I want to use TF to train each layer based on its own local loss. It means when training the current layer, the previous layers' parameters should be fixed.

So my question is how to construct one or, if necessary, several graphs to reaize the above idea?

Any comments and suggestions would be appreciated. Thanks.

Solution

  • You can try using .compute_gradients(loss, var_list) method of the optimizer. 

    wrt_layer1 = opt.compute_gradients ( loss1 , < list of variables >)
wrt_layer2 = opt.compute_gradients ( loss2 , < list of variables >)
....

    Each returns a list of tuples (gradient, variable).
    Concat each of these lists 

    final_grads = wrt_layer1 + wrt_layer2 + .....

    and finally use .apply_gradients(grads_and_vars) 

    opt.apply_gradients (final_grads)

    I guess that might work for you.