I am currently using Encog 3.3 (Java implementation) in order to train a 2^N-10-10-2 MLP network with BPROP and RPROP (in two separate scenarios).
I have divided my data set in 40/20/40 (training/validation/test) randomized sub-sets.
Since I have a validation set, I can add Encog's EarlyStoppingStrategy to my training.
It happens that I am a bit confused about how to set these 3 parameters:
The method is often getting stuck at local minima.
The training method is configured as follows:
// configure learning rule (backpropagation)
Backpropagation train = new Backpropagation(network, trainingSet, 1e-6, 0.9);
// configure training strategy
EarlyStoppingStrategy early = new EarlyStoppingStrategy(validationSet, trainingSet, trainingSet.size(), 1, 0.01);
train.addStrategy(early);
Is there a proper/recommended way to set these parameters?
If your training is getting stuck at local minima, I don't think your problem is with the EarlyStoppingStrategy. Instead, your problem has to do with the Momentum of the back propagation algorithm.
The EarlyStoppingStrategy stops the training when the cross validation error increases which you don't want to happen as the training error decreases as this means you are "overtraining". However, the EarlyStoppingStrategy has nothing to do with reducing the global training error of your training process.
It is very hard to set both the "Learning Rate" and the Momentum for the Back Propagation algorithm.
I would suggest that you first try the ResilientPropagation algorithm, as it is easier to use, and does not require you to set these parameters. You might also consider the QuickPropagation training algorithm.
If that is not an options, then I would suggest you use "Smart Momentum" with back propagation as shown below:
Backpropagation train = new Backpropagation(network, trainingSet);
train.addStrategy(new SmartMomentum());
SmartMomentum uses the error gradient from the previous training iteration as well as the error gradient from the current training iteration to help the training process avoid local minima.
If you still have problems getting your global error down to the desired level, you might also try adding the SmartLearningRate strategy, found here.