I want to compare two "Evolution Startegies" using SR (Sucess Rate) and AES (average number of evaluations to a solution). I am not sure of how "AES" works...
Let's say, I have 10 runs of the algorithm and the optimum is at value 0.0:
Run 1: 0.1
Run 2: 0.8
Run 3: NAN
Run 4: 1.5
Run 5: 0.4
Run 6: 0.3
Run 7: NAN
Run 8: 1.9
Run 9: 1.1
Run 10: 0.5
I understand that if I choose the limit of 1.0, the succes rate is: 5/10.
Questions:
1- Is the SR value right?
2- What is the value of the AES?
3- Do I have to use the same limit value (1.0 in thus example) for the SR and the AES?
4- To calculate the AES, do I need to know in which iteration of each run the value was lower than 1.0 (for example)? Could you provide an example?
Thanks in advance
Success rate is the ratio of number of runs that an specific algorithm could reach to an expected accuracy to the total number of runs. It means that: SR = (number of successful runs/total number of runs). To do this, you first need to determine a threshold for errors. This threshold depends on how hard the problem is. An example can be found in Kordestani, J. K., A. Ahmadi, and M. R. Meybodi. 2014. “An Improved Differential Evolution Algorithm using Learning Automata and Population Topologies.” Applied Intelligence 41 (4): 1150–1169., where 1e-10 has been considered as an expected accuracy for some easy to optimize functions and 1e-6 and 1e-2 have been considered for some harder functions. Average number of evaluations to a solution is the number of evaluations spended to reach the considered accuracy which are averaged over various runs. Following is an example to further examin SR and AES. Assume that we run an algorithm for optimizing a mathematical function 5 times (atleast 30 times should be performed to reach a conclusion about the performance of different algorithms as suggested in different papers however for the sake of an example we consider 5 runs). Moreover, the termination condition is reaching 10000 function evaluation. The followings are the error of the best solution found(err), successful run(ea), and the evaluation number where the algorithm reaches to the expected accuracy(FE).
run err ea FE Successful run
1 1.35e-3 1e-2 8800 YES
2 1.05e-1 1e-2 - NO
3 2.95e-4 1e-2 6000 YES
4 5.55e-8 1e-2 4200 YES
5 9.98e-0 1e-2 - NO
Considering the above table, SR = (3/5) = 0.6 or 60%. The AES for successful runs is also calculated as (8800+6000+4200)/3