I'm getting started working with neural networks. I have adapted the XOR example provided for my own purposes, but when I run it the error never changes.
The function I'm trying to approximate takes 4 doubles and outputs 1 double, the 4 inputs are always positive, the output can be negative or positive (majority is positive). For starters I am using 50 records to train the data.
Working XOR (The error goes down with each iteration)
public static double[][] XORInput = {
new[] {0.0, 0.0},
new[] {1.0, 0.0},
new[] {0.0, 1.0},
new[] {1.0, 1.0}
};
public static double[][] XORIdeal = {
new[] {0.0},
new[] {1.0},
new[] {1.0},
new[] {0.0}
};
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(null, true, 2));
network.AddLayer(new BasicLayer(new ActivationSigmoid(), true, 3));
network.AddLayer(new BasicLayer(new ActivationSigmoid(), false, 1));
network.Structure.FinalizeStructure();
network.Reset();
IMLDataSet trainingData = new BasicMLDataSet(XORInput, XORIdeal);
IMLTrain train = new ResilientPropagation(Network, trainingData);
Not Working (the error never goes down):
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(null, true, 4));
network.AddLayer(new BasicLayer(new ActivationSigmoid(), true, 6));
network.AddLayer(new BasicLayer(new ActivationSigmoid(), false, 1));
network.Structure.FinalizeStructure();
network.Reset();
IMLDataSet trainingData = new BasicMLDataSet(myInput, myExpectedOutput);
IMLTrain train = new ResilientPropagation(Network, trainingData);
A few sample records of the training data:
Input:
2.54, 3.15, 3.4, 1.73
5.3, 1.78, 3.9, 2.04
1.71, 5.4, 4.3, 2.26
1.62, 6.4, 4, 1.89
1.45, 8.4, 5.2, 2.14
Output:
5.59
11.05
6.89
10.4
-0.56
I believe that the problem is that activation function isn't firing. I thought it might be because ActivationSigmoid() is inappropriate for this problem, but I have tried ActivationTANH() with the exact same results.
The problem is that my values weren't being normalised.
To work with the activation functions all of your inputs and outputs must be between 1 and 0 (ActivationSigma) and -1 and 1 (ActivationTANH). You need some function to normalise your values to the range that they need to be.
This link was of great help to me: