Calculation weigths in combine classifier outputs

Question

I have 3-class classification problem and I have ensemble with 4 classifiers. Each classifier returns supports for each class. Now I want to combine this outputs using weighted average with c*L weights (c=3, L=4), but I don't know how can I calculate weights?

For example, I have dataset like this:

D1,1 | 0.85 0.01 0.34 0.44 0.08 0.89 0.01 0.35 0.97 0.04
D1,2 | 0.67 0.03 0.13 0.58 0.06 0.48 0.60 0.59 0.03 0.28
D1,3 | 0.52 0.08 0.50 0.87 0.88 0.01 0.95 0.58 0.83 0.65
D2,1 | 0.29 0.26 0.85 0.46 0.23 0.62 0.09 0.66 0.83 0.48
D2,2 | 0.70 0.02 0.38 0.43 0.86 0.23 0.03 0.64 0.04 0.97
D2,3 | 0.38 0.42 0.69 0.74 0.71 0.52 0.88 0.43 0.54 0.74
D3,1 | 0.56 0.34 0.62 0.46 0.87 0.72 0.24 0.13 0.94 0.56
D3,2 | 0.88 0.54 0.45 0.86 0.93 0.60 0.00 0.75 0.32 0.29
D3,3 | 0.84 0.92 0.47 0.46 0.13 0.58 0.81 0.24 0.80 0.25
D4,1 | 0.89 0.29 0.94 0.49 0.39 0.43 0.14 0.65 0.60 0.88
D4,2 | 0.93 0.33 0.08 0.48 0.98 0.24 0.87 0.85 0.78 0.44
D4,3 | 0.81 0.85 0.27 0.22 0.64 0.42 0.09 0.08 0.79 0.81
TARGET    1    3    1    3    3    1    2    2    2    3

Di,j is support of i-th classifier for j-th class. TARGET is truth class of observation.

Answer 1

I do this in this way, but I don't know it's good. In my opinion it returns good results, therefore I think that it's good. Somebody can check this ?

%% Prepare environment
clearvars;
close all;
format compact;
clc;

%% Code here
nObservations = 10;
nClasses = 3; % c
nClassifiers = 4; % L

data = [0.85 0.01 0.34 0.44 0.08 0.89 0.01 0.35 0.97 0.04;
    0.67 0.03 0.13 0.58 0.06 0.48 0.60 0.59 0.03 0.28;
    0.52 0.08 0.50 0.87 0.88 0.01 0.95 0.58 0.83 0.65;
    0.29 0.26 0.85 0.46 0.23 0.62 0.09 0.66 0.83 0.48;
    0.70 0.02 0.38 0.43 0.86 0.23 0.03 0.64 0.04 0.97;
    0.38 0.42 0.69 0.74 0.71 0.52 0.88 0.43 0.54 0.74;
    0.56 0.34 0.62 0.46 0.87 0.72 0.24 0.13 0.94 0.56;
    0.88 0.54 0.45 0.86 0.93 0.60 0.00 0.75 0.32 0.29;
    0.84 0.92 0.47 0.46 0.13 0.58 0.81 0.24 0.80 0.25;
    0.89 0.29 0.94 0.49 0.39 0.43 0.14 0.65 0.60 0.88;
    0.93 0.33 0.08 0.48 0.98 0.24 0.87 0.85 0.78 0.44;
    0.81 0.85 0.27 0.22 0.64 0.42 0.09 0.08 0.79 0.81];
data = data';
targets = [1 3 1 3 3 1 2 2 2 3]';

w = zeros(nClassifiers, nClasses);
for iClass = 1:nClasses
    isMemberOfIthClass = zeros(nObservations, 1);
    isMemberOfIthClass(targets == iClass) = 1;
    c = cov(bsxfun(@minus, isMemberOfIthClass, data(:, ((1:nClassifiers)-1)*nClasses+iClass)));
    w(:, iClass) = c \ ones(nClassifiers, 1) / (ones(1, nClassifiers) / c * ones(nClassifiers, 1));
end

mi = zeros(nObservations, nClasses);
for iClass = 1:nClasses
    mi(:, iClass) = data(:, ((1:nClassifiers)-1)*nClasses+iClass)*w(:, iClass);
end

[~, ind] = max(mi, [], 2);
[mi ind targets]