I have done some extensive research on SA (Simulated Annealing). Even so, I am having a hard time understanding how to find input parameters.
In all of my research it seems you just start with a shot in the dark and adjust from there. That seems extremely inefficient and unlikely to yield quality results.
How do you find what parameters to use in a SA training algorithm using Encog (such as: Start Temp, Stop Temp, Cycles), in order to produce an efficient and quality result?
Simulated annealing is a process where a vector of values are adjusted to improve the score of an objective function. If you are using Encog to train a neural network with simulated annealing, then the vector you are trying to adjust/optimize is the weights of the neural network. The objective function is simply what error the neural network gets for a current set of weights when compared against the desired outputs for a given dataset. Consider the random walk, which is one of the most simple optimization methods. Here you choose one weight and either increase it or decrease it (entirely random). If the change to the weight lowered the error, then that weight change becomes permanent and the cycle repeats on a new weight. If the error does not improve, the weight change is undone and a new weight is selected. You can think of random walk as randomly walking between weight configurations and only keeping a new position if the error improves.
Simulated annealing is very similar but it accepts the fact that sometimes you must accept a new position that has a worse error, in order to move forward to better error. The old sometimes you must take one step backwards to take two steps forward. The temperature range simply specifies how likely simulated annealing is to accept a bad move. Higher temperatures are more likely than lower ones. The temperature starts high and goes lower. At lower temperatures simulated annealing is essentially a random walk. The cycles parameter specifies how many times the algorithm attempts to move to a better location.