When to start using the selection set in a Back Propagation Neural Network?

Question

Beginner on ANNs:

I am implementing a back propagation neural network to predict the price of gold. I know that I have to split my data into training data, selection data and test data.

However I unsure How to go on about using these sets of data. At first I was training the data network with my training set then after it's trained I am getting a number of inputs to my network from the test set and comparing the output.

I'm not sure if I'm doing this right and were does the selection set come in ?

thanks in advance!

Answer 1

The general idea is:

1. Train the network for a little while on the training set.
2. Evaluate the network on a second set, often called the validation set. Probably what you're calling the selection set.
3. Train the network a little more on the training set.
4. Evaluate the new network on the selection set again.
5. Which did better, the old network or the new network? If the new network is better, we're still getting some use out of training, so goto 3. If the new network is worse, more training will probably only hurt. Use the previously version of the network, since it did better.

In this way, you can tell when to stop training.

One easy modification to this is to always keep track of the best network seen so far, and we only stop training when we see some number (say, three) of training attempts that do worse in a row.

The third set, the test set, is necessary because the selection set is, if indirectly, involved in the training process. Final evaluation must be done on data that was not used at all during training.

This sort of thing is sufficient for simple experiments, but in general you'll want to use cross-validation to get a better idea of your system's performance.