Search code examples
machine-learningdeep-learningbackpropagation

small learning rate vs big learning rate

I'm new in ML. While I was reading about backpropagation Suddenly, I have question.
In backpropagation learning of a neural network,
Do we should start with a small learning rate and slowly increase it during the learning process? or
Do we should start with big learning rate and slowly reduce it during the learning process?

Which one is correct?

Solution

  • Generally, the second one it correct - Think about this in this way - big learning rate means that you roughly search for the best area in the space. Then, with a small learning you tune the weights to find the best value.

    If you would use constant big learning rate you would "jump" around the minimum point. If you would use constant small learning rate it would take a lot of time to converge. That`s why learning rate decaying is a good idea.

    enter image description here

    Having said that, there are a few more advanced tricks for learning rate scheduling, that are not monotonically decreasing the learning rate.