Search code examples
neural-network

In neural networks, do the weights of bias terms get updated in backpropagation?

In neural networks, I am trying to code it out, but got stuck on one part. Do the weights of bias terms get updated in backpropagation? I am following the algorithm here http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ and they seem to not update it, and they chose some arbitrary values for bias term weights. Does that mean the outputs of bias terms are 1, and its weights should not change?

Solution

  • Generally: yes, the bias weights should be updated and included in training just as any other weight in the NN (also in backpropagation).

    In the example you've posted, the bias b1 is added to both neurons the hidden layer, and bias b2 to both neutrons in the output layer

    Hidden layer:
    h1 = i1*w1 + i2*w2 + 1*b1
    h2 = i1*w3 + i2*w4 + 1*b1

Output layer:
    o1 = h1*w5 + h2*w6 + 1*b2
    o2 = i2*w7 + h2*w8 + 1*b2

    With initial, and in this example, fixed, biases

    b1 = 0.35
b2 = 0.60

    This means that the biases for hidden neutrons is always exactly 0.35, and for output neutrons exactly 0.60. This, however, is not usual practice, as you want to train your NN to find "good" biases just as much you want it to train to find good weights.

    Note also, in the comments of the link you provided, that another user has asked why biases are not changed, and the author has replied, quote:

    "Hey, in the tutorials I went through they didn’t update the bias which is why I didn’t include it here."

    This lack of a specific "why" possibly implies that the author of this example/tutorial is, however well-versed, no expert at the subject of NN, so you shouldn't put to much weight (no pun intended...) into the fast that the biases are not changed in this specific example.

    If you really want to dig into a sound and thorough covering of NN in the context of back propagation, I would rather recommend you Michael Nielsen excellent book on NN and deep learning, specifically for this subject, Chapter 2. Note that the bias weights, here, are treated just as weights for neuron-neuron data transfer.

    Michael is a Google Researcher with numerous publicised articles in the subject of advanced NN and deep learning.

    At the heart of backpropagation is an expression for the partial derivative ∂C/∂w of the cost function C with respect to any weight w (or bias b) in the network. The expression tells us how quickly the cost changes when we change the weights and biases.