Search code examples
matlabneural-networklinear-regressionbackpropagationperceptron

Any Ideas for Predicting Multiple Linear Regression Coefficients by using Neural Networks (ANN)?

In case, there are 2 inputs (X1 and X2) and 1 target output (t) to be estimated by neural network (each nodes has 6 samples): 

X1 = [2.765405915 2.403146899 1.843932529 1.321474515 0.916837222 1.251301467]; 

X2 = [84870 363024 983062 1352580 804723 845200];

t = [-0.12685144347197 -0.19172223428950 -0.29330584684934 -0.35078062276141 0.03826908777226 0.06633047875487];

I was trying to find the best fit of t prediction by using multiple linear regression (Ordinary Least Squares or OLS) manually and the outcomes were pretty good.

I intend to find the a, b, c (regression coefficients) from this equation:

t = a + b*X1 + c*X2

Since the equation is the basic form of multiple linear regression equation with two regressors, of course I could find the value of a, b, c by doing the OLS.

The problem is: I've tried to find the regression coefficients by using neural network (with MATLAB nftool and train it by Levenberg-Marquardt Backpropagation or lmtrain) but have no idea how to find them, though the outcomes were showing less error than the OLS.

Then, several questions that comes:

  1. Is it possible to find the regression coefficient by using neural network?
  2. If it's possible, What kind of ANN algorithm that could solve this kind of problem and how to build it manually?

If you have any ideas how to solve it, please help. I really need your help!

This is the script that generated by MATLAB nftool that I used to fit the output estimation:

% Solve an Input-Output Fitting problem with a Neural Network
% Script generated by NFTOOL
% Created Fri Jun 05 06:26:36 ICT 2015
%
% This script assumes these variables are defined:
%
%   x - input data.
%   t - target data.
x = [2.765405915 2.403146899 1.843932529 1.321474515 0.916837222 1.251301467; 84870 363024 983062 1352580 804723 845200];
t = [-0.12685144347197 -0.19172223428950 -0.29330584684934 -0.35078062276141 0.03826908777226 0.06633047875487];

inputs = x;
targets = t;

% Create a Fitting Network
hiddenLayerSize = 10;
net = fitnet(hiddenLayerSize);


% Setup Division of Data for Training, Validation, Testing
net.divideParam.trainRatio = 90/100;
net.divideParam.valRatio = 5/100;
net.divideParam.testRatio = 5/100;


% Train the Network
[net,tr] = train(net,inputs,targets);

% Test the Network
outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs)

% View the Network
view(net)

% Plots
% Uncomment these lines to enable various plots.
%figure, plotperform(tr)
%figure, plottrainstate(tr)
%figure, plotfit(net,inputs,targets)
%figure, plotregression(targets,outputs)
%figure, ploterrhist(errors)
Solution

  • A neural network will generally not find or encode a formula like t = a + b*X1 + c*X2, unless you built a really simple one with no hidden layers and linear output. If you did then you could read the values [a,b,c] from the weights attached to bias, input 1 and input 2. However, such a network offers no advantage over linear regression (essentially it is linear regression using NN tools to build it and comparatively slow gradient descent to find the lowest least-square error when it can be done in a single pass in OLS).

    What you have built instead is a more complex non-linear function. Most likely the error is low because you have over-fit your data, which is very easy to do with a neural net. With your input data as shown, it should be possible to get an training error of 0, but that is not as good as it seems - it just means the neural network has found a complex surface that connects all your examples, which is probably of limited use as a predictive model.