I am writing a program in Matlab with the task of using 5 different models to fit 8 parameters and output their FVAL in order to calculate AIC and BIC to obtain the best model out of the 5. My problem is that although the parameter fitted by the different models are completely different I get the same FVAL score which makes no sense to me.
My task has 234 data points and 8 parameters in total but every data point uses only 2 parameters. The answer is binary 1 or 0. I have a 234x1 double variable called 'results' which is the score i want to predict, either 1 or 0 for every data point. Another 234x1 double variable called 'tParamVal' which the first parameter out of the 2 that was used, this is scored as 1, 2, 3, or 4. Lastly, 234x1 double variable caleld 'sParamVal' which is the second parameter that was used, this is scored as 5, 6, 7, or 8. So for every data point out of the 234, I have param1 (for example, 3), param2 (for example 8), results (for example 1).
I defined my models like that:
null = @(params, x) mean(results); % null model.
multiplicative = @(params, x) params(x(1)) * params(x(2)); % multiplicative model
minimalism = @(params, x) min(params(x(1)), params(x(2))); % minimum model.
For the objective function, I used negative log likelihood defined as:
objective_null = @(params) -mean((results .* log(null(params, ...
[tParamVal, sParamVal]) + 1e-10)) + ((1 - results) .* ...
log(1 - null(params, [tParamVal, sParamVal]) + 1e-10)));
objective_mul = @(params) -mean((results .* log(multiplicative(params, ...
[tParamVal, sParamVal]) + 1e-10)) + ((1 - results) .* ...
log(1 - multiplicative(params, [tParamVal, sParamVal]) + 1e-10)));
objective_min = @(params) -mean((results .* log(minimalism(params, ...
[tParamVal, sParamVal]) + 1e-10)) + ((1 - results) .* ...
log(1 - minimalism(params, [tParamVal, sParamVal]) + 1e-10)));
Those are the 3 basic models and I added another 2 models that have a constraint with the following rules: The scores of 1st parameter > 2nd param > 3rd param > 4th param and another rule: 5th param > 6th > 7th > 8th. This is how I defined the constraint:
constraint = @(params) [params(1) - params(2), params(2) - params(3), ...
params(3) - params(4), params(5) - params(6), ...
params(6) - params(7), params(7) - params(8)];
The objective functions for the last 2 models with the constraints are defined as:
penalized_objective_mul = @(params) objective_mul(params) + ...
sum(min(0, constraint(params)).^2);
penalized_objective_min = @(params) objective_min(params) + ...
sum(min(0, constraint(params)).^2);
For the BADS optimization algorithm you need to provide a starting point x0:
x0 = [0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5]; % Starting point
And hard lower bound (lb) and upper bound (ub):
lb = [0 0 0 0 0 0 0 0]; % Lower bounds
ub = [1 1 1 1 1 1 1 1]; % Upper bounds
To obtain the fitted paramters and fval I run the following lines for the 5 models:
% null model
[null_fit, null_fval] = bads(objective_null, x0, lb, ub);
nullStruct.(subject) = [null_fit, null_fval];
% multiplicative model
[mul_fit, mul_fval] = bads(objective_mul, x0, lb, ub);
mulStruct.(subject) = [mul_fit, mul_fval];
% multiplicative with constraint model
[mulConst_fit, mulConst_fval] = bads(penalized_objective_mul, x0, lb, ub);
mulConstStruct.(subject) = [mulConst_fit, mulConst_fval];
% minimalism model
[min_fit, min_fval] = bads(objective_min, x0, lb, ub);
minStruct.(subject) = [min_fit, min_fval];
% minimalism with constraint model
[minConst_fit, minConst_fval] = bads(penalized_objective_min, x0, lb, ub);
minConstStruct.(subject) = [minConst_fit, minConst_fval];
This is all being run in a for loop for each 'subject'.
For each subject I get a different FVAL score but within the same subject and between the different models i get the same FVAL score. Worth noting, while the FVAL is the same, the different models output different parameters that they fitted.
I expect the problem is in the objective function which i tried to change and review a few times but without success.
Thanks in advance.
The problem was in the models functions.
Corrected models functions:
multiplicative = @(params, x) (params(x(:,1)) .* params(x(:,2)))';
minimalism = @(params, x) (min(params(x(:,1)), params(x(:,2))))';