Search code examples
neural-networkbackpropagation

Clarification on back propagation

As I usually do when isolated at home for too long, I was thinking about back-propagation.

If my thought process is correct, for computing the weights update we never actually need to compute the cost. We only ever need to compute the derivative of the cost.

Is this correct?

I imagine that the only reason to compute the Cost would be to check if the network is actually learning.

I really believe I am correct, but by checking on the internet no one seems to make this observation. So maybe I am wrong. If I am, I have a deep misunderstanding of backpropagation that I need to fix.

Solution

  • You are correct.

    The cost function is what tells you how much the solution costs. The gradient is what carries the information about how to make it cost less.

    You could shift the cost with any constant addition or subtraction and it wouldn't make a difference, because there is no way to make that part of the cost go down.