Consider I am writing a program to objectively select a winner in a competition. There are 'n' human judges secretly assigning a 1st, 2nd, 3rd position ranking to the top three candidates from a pool of 'm' candidates.
The program must then go through the judges decisions, and based on weights assigned to 1st place, 2nd place and 3rd place, each candidate will be rated based on the number of 1st, 2nd, and 3rd place votes they received, multiplied by the appropriate rating for each finishing position.
However, at the start, the program has no idea of what weights are appropriate, so I have created an automated "program" that is intended to "discover" the proper weights based on how the judges would pick the winner from a hypothetical situation.
I present a table where the horizontal axis contains the finishing position, and the judges' codes (e.g. Judge W, Judge X, Judge Y, Judge Z). The vertical axis has three rows (1st place, 2nd place, 3rd place), and at the intersection of each Judge/Row, I have randomly generated a candidate ID (from the set A through F).
After rendering the table, I then ask the judge who THEY would have chosen as the winner (the judge has the option to PASS if there is not sufficient information to choose).
After the judges run through an appropriate number of scenarios, I wish to now take the results of the various runs and use that information to determine the "best fit" for the weighting of 1st, 2nd, and 3rd positions.
Let's say one of the hypothetical grids looks like this:
<table border="1"><tbody><tr><th>Position</th><th>Judge 'W'</th><th>Judge 'X'</th><th>Judge 'Y'</th><th>Judge 'Z'</th></tr><tr><td>1st</td><td><center>A</center></td><td><center>F</center></td><td><center>C</center></td><td><center>B</center></td></tr><tr><td>2nd</td><td><center>D</center></td><td><center>B</center></td><td><center>E</center></td><td><center>D</center></td></tr><tr><td>3rd</td><td><center>C</center></td><td><center>E</center></td><td><center>B</center></td><td><center>C</center></td></tr></tbody></table>and the human judge picks candidate "B" as the winner. My program should react by calculating the (w1 + w2 + w3) > (w1 + 2w3) (i.e. B better than C) and (w1 + w2 + w3) > (2 w2) (i.e. B better than D), etc.
From these various algebraic comparisons, over a number of "hypothetical scenarios", I want to be able to calculate the optimum values for w1, w2 and w3. And then, at some point when there is enough "good" data, I want to be able to use these "discovered" weights to go back over the training data an identify areas where perhaps the human judges were mistaken.
I am using PHP as the programming language and don't know which functions or possible existing libraries are appropriate to solve this kind of "fuzzy" equation.
I'm looking for some direction to help me tackle this problem.
Thank you for your assistance.
For the winning candidate count how many times he appears in each position, then do the same for all the other candidates. Then write the following formula for each candidate:
GoodForJ=w1*nw1+w2*nw2+w3*nw3>w1*nj1+w2*nj2+w3*nj3
Where nw1-3 are the times the winner appears in each position and nj1-3 are the times the j candidate appears in each position. If goodForJ is true for all the candidates this means that the tuple of weights is good. Now you just have to try a bounch of weights combinations and find out which one fits. Trying all combinations of weights between 1 and 10 requires 1000 iterations. To make things a bit fuzzier, for each try you could count how many timrs goodForJ is true and choose the weights that produces the highest score.