How do we define the bad learning rate in neural network?

Question

I am trying to define the proper definition of the bad learning rate in the neural network as follows:

Bad learning rate in the neural network is when you assigned the learning rate too low or too high, with too low learning rate the network would take too much time to train but with too high learning rate the network would change too quick which might result in output.

Any suggestion would be much appreciated.

Answer 1

I believe that an efficient learning rate(alpha) depends on the data. The points that you have mentioned are absolutely correct with regards to inefficient learning rates. So, there is no hard and fast rule for selecting alpha. Let me enumerate the steps that I take while deciding alpha :

1. You would obviously need a big alpha so that your model learns quickly
2. Also take note that big alpha can lead to overshooting the minima and hence your hypothesis will not converge
3. To take care of this, you can go for learning rate decay. This decreases your learning rate as you approach minima and slow down learning so that your model doesn't overshoot.

Couple of ways to do this:

• Step decay
• exponential decay
• linear decay

You can select either and then train your model. Having said that, let me point out that it still requires some trial and error from your side until you get the optimum result.