Search code examples
pythontheanolasagne

Lasagne / Theano gradient values

I'm currently working on recurrent neural nets using Lasagne / Theano.

While training, updates are calculated using Theano's symbolic gradient.

grads = theano.grad(loss_or_grads, params)

While the gradient expression is perfectly fine in general, I'm also interested in the gradient values in order to monitor training.

My question now is if there is a built-in method to also get gradient values, which I haven't found so far, or if I'll have to do it myself.

Thanks in advance

Solution

  • I'm not aware of any lasagne function to evaluate the gradient, but you can get it yourself with simple theano function.

    Say we have the following theano variables:

    • inputs = Inputs to the network
    • targets = Target outputs of the network
    • loss = Value of the loss function, defined as a function of network outputs and targets
    • l_hid = Recurrent layer of the network, type lasagne.layers.RecurrentLayer

    Say we're interested in the gradient of the loss function w.r.t. the recurrent weights:

    grad = theano.grad(loss, l_hid.W_hid_to_hid)

    Define a theano function to get a numerical value for the gradient

    get_grad = theano.function([inputs, targets], grad)

    Now, just call get_grad for any value of the inputs and targets (e.g. the current minibatch). get_grad() doesn't need to be passed the value of the weights because they're stored as a theano shared variable.