A chess rating system for evolutionary algorithms: A new method for the comparison and ranking of evolutionary algorithms

Information Sciences

Volumn 277, Issue , 2014, Pages 656-679

  Veček, Niki ^a   Mernik, Marjan ^a   Črepinšek, Matej ^a  

^a UNIVERSITY OF MARIBOR   (Slovenia)

Author keywords

Chess rating; Computational experiment; Evolutionary algorithm; Null hypothesis significance testing; Ranking

Indexed keywords

EXPERIMENTS; RATING;

BENCHMARK SUITES; BRADLEY-TERRY MODELS; COMPUTATIONAL EXPERIMENT; OBJECTIVE FUNCTIONS; PAIR-WISE COMPARISON; RANKING; SIGNIFICANCE TESTING; STATISTICAL SIGNIFICANCE TEST;

EVOLUTIONARY ALGORITHMS;

EID: 84901805253     PISSN: 00200255     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ins.2014.02.154     Document Type: Article

References (91)

1. M. Ali, M. Pant, and A. Abraham Improving differential algorithm by synergizing different improvement mechanisms ACM Trans. Auton. Adapt. Syst. 7 2 2012 20:1 20:32
2. H.M. Alizadeh, A.A. Khamseh, and S.F. Ghomi Fuzzy hypothesis testing with vague data using likelihood ratio test Soft Comput. 17 2013 1629 1641
3. M. Altman, J. Gill, and M.P. McDonald Numerical Issues in Statistical Computing for the Social Scientist vol. 508 2004 John Wiley & Sons
4. T. Bäck, D.B. Fogel, and Z. Michalewicz Handbook of Evolutionary Computations 1996 Oxford University Press
5. R.S. Barr, B.L. Golden, J.P. Kelly, M.G.C. Resende, and W.R. Stewart Jr. Designing and reporting on computational experiments with heuristic methods J. Metaheur. 1 1995 9 32
6. T. Bartz-Beielstein Experimental Research in Evolutionary Computation 2006 Springer
7. V.W. Berger A Priori v Post Hoc Testing Encyclopedia of Statistics in Behavioral Science 2005 John Wiley & Sons
8. C. Blum, and A. Roli Metaheuristics in combinatorial optimization: overview and conceptual comparison ACM Comput. Surv. 35 3 2003 268 308
9. C. Blum, J. Puchinger, G.A. Raidl, and A. Roli Hybrid metaheuristics in combinatorial optimization: a survey Appl. Soft Comput. 11 6 2011 4135 4151
10. R.A. Bradley, and M.E. Terry Rank analysis of incomplete block designs: I. The method of paired comparisons Biometrika 39 3/4 1952 324 345
11. J. Brest, S. Greiner, B. Bošković, M. Mernik, and V. Žumer Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems IEEE Trans. Evolution. Comput. 10 6 2006 646 657
12. J. Brest, P. Korošec, J. Šilc, A. Zamuda, B. Bošković, and M.S. Maučec Differential evolution and differential ant-stigmergy on dynamic optimisation problems Int. J. Syst. Sci. 44 4 2013 663 679
13. F. Caraffini, F. Neri, G. Iacca, and A. Mol Parallel memetic structures Inform. Sci. 227 2013 60 82
14. J. Cohen The earth is round (p <. 05) Am. Psychol. 49 12 1994 997 1003
15. M. Črepinšek, S.H. Liu, and L. Mernik A note on teaching-learning-based optimization algorithm Inform. Sci. 212 2012 79 93
16. M. Črepinšek, S.H. Liu, M. Mernik, Exploration and exploitation in evolutionary algorithms: a survey, ACM Comput. Surv. 45(3) (2013) 35.
17. S. Das, and P.N. Suganthan Differential evolution: a survey of the state-of-the-art IEEE Trans. Evolut. Comput. 15 1 2011 4 31
18. J. Demšar Statistical comparisons of classifiers over multiple data sets J. Mach. Learn. Res. 7 2006 1 30
19. J. Derrac, S. Garcia, D. Molina, and F. Herrera A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms Swarm Evolut. Comput. 1 1 2011 3 18
20. M. Dowell, and P. Jarratt A modified regula falsi method for computing the root of an equation BIT Numer. Math. 11 2 1971 168 174
21. O.J. Dunn Multiple comparisons among means J. Am. Statist. Assoc. 56 1961 52 64
22. C.W. Dunnett A multiple comparison procedure for comparing several treatments with a control J. Am. Statist. Assoc. 50 1980 1096 1121
23. T. Dybå, V.B. Kampenes, and D.I. Sjøberg A systematic review of statistical power in software engineering experiments Inform. Softw. Technol. 48 8 2006 745 755
24. Efficient Java Matrix Library < http://code.google.com/p/efficient- java-matrix-library >, 2013.
25. Á.E. Eiben, M. Jelasity, A critical note on experimental research methodology in EC, in: Proceedings of the 2002 Congress (CEC'2002), vol. 1, 2002, pp. 582-587.
26. Á.E. Eiben, and J.E. Smith Introduction to Evolutionary Computing 2008 Springer
27. A.E. Elo The Rating of Chessplayers, Past and Present vol. 3 1978 Batsford
28. Evolutionary Algorithms Rating System < http://earatingsystem.appspot. com/ > (version 1), 2013.
29. Evolutionary Algorithms Rating System (Github) < https://github.com/ matejxxx/EARS > (version 1), 2013.
30. Experimental results for CRS4EAs (raw data) < http://lpm.uni-mb.si/ crepinsek/dataINS.txt >, 2013.
31. A.P. Field, G. Hole, How to Design and Report Experiments, Sage publications, 2003.
32. H. Finner On a monotonicity problem in step-down multiple test procedures J. Am. Statist. Assoc. 88 1993 920 923
33. R.A. Fisher Statistical Methods for Research Workers 1925 Oliver & Boyd
34. R.A. Fisher The Design of Experiments 1935 Oliver & Boyd
35. R.A. Fisher Statistical Methods and Scientific Inference second ed. 1959 Hafner Publishing Co.
36. R.A. Fisher Statistical Methods, Experimental Design, and Scientific Inference: A Re-Issue of Statistical Methods for Research Workers, The Design of Experiments, and Statistical Methods and Scientific Inference 1995 Oxford University Press
37. M. Friedman The use of ranks to avoid the assumption of normality implicit in the analysis of variance J. Am. Statist. Assoc. 326 1937 75 701
38. M. Friedman A comparison of alternative tests of significance for the problem of m rankings Ann. Math. Statist. 11 1940 86 92
39. S. García, A. Fernández, J. Luengo, and F. Herrera Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power Inform. Sci. 180 10 2010 2044 2064
40. S. García, and F. Herrera An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons J. Mach. Learn. Res. 9 2008 2677 2694
41. J. Gill The insignificance of null hypothesis significance testing Polit. Res. Quart. 52 3 1999 647 674
42. M.E. Glickman A comprehensive guide to chess ratings Am. Chess J. 3 1995 59 102
43. M.E. Glickman The Glicko System 1995 Boston University
44. M.E. Glickman Parameter estimation in large dynamic paired comparison experiments J. Roy. Statist. Soc.: Ser. C (Appl. Statist.) 48 3 1999 377 394
45. M.E. Glickman Dynamic paired comparison models with stochastic variances J. Appl. Statist. 28 6 2001 673 689
46. M.E. Glickman Example of the Glicko-2 System 2012 Boston University
47. W. Gong, and Z. Cai Differential evolution with ranking-based mutation operators IEEE Trans. Cybernet. 1-16 2013
48. N. Hansen The CMA Evolution Strategy: A Comparing Review Towards A New Evolutionary Computation 2006 Springer pp. 75-102
49. L.L. Harlow, and S.A. Mulaik What If There Were No Significance Tests? 1997 Lawrence Erlbaum Associates
50. K. Harkness Official Chess Handbook 1967 D. McKay Company
51. Y. Hochberg A sharper Bonferroni procedure for multiple tests of significance Biometrika 75 1988 800 803
52. J.L. Hodges, and E.L. Lehmann Rank methods for combination of independent experiments in analysis of variance The Ann. Math. Statist. 33 2 1962 482 497
53. B.S. Holland, and M.D. Copenhaver An improved sequentially rejective Bonferroni test procedure Biometrics 43 1987 417 423
54. S. Holm A simple sequentially rejective multiple test procedure Scand. J. Statist. 6 1979 65 70
55. G. Hommel A stagewise rejective multiple test procedure based on a modified Bonferroni test Biometrika 75 1988 383 386
56. T. Hoshino Bayesian significance testing and multiple comparisons from MCMC outputs Comput. Statist. Data Anal. 52 7 2008 3543 3559
57. G. Iacca, F. Neri, E. Mininno, Y.S. Ong, and M.H. Lim Ockham's razor in memetic computing: three stage optimal memetic exploration Inform. Sci. 188 2012 17 43
58. R.L. Iman, and J.M. Davenport Approximations of the critical region of the Friedman statistic Commun. Statist. - Theory Meth. 9 6 1980 571 595
59. G. Jia, Y. Wang, Z. Cai, and Y. Jin An improved (+< lambda>)-constrained differential evolution for constrained optimization Inform. Sci. 222 2013 302 322
60. D. Karaboga, and B. Basturk On the performance of artificial bee colony (ABC) algorithm Appl. Soft Comput. 8 1 2008 687 697
61. B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg Preliminary guidelines for empirical research in software engineering IEEE Trans. Softw. Eng. 28 8 2002 721 734
62. A. Knezevic, StatNews #73: Overlapping Confidence Intervals and Statistical Significance, Cornell Statistical Consulting Unit, 2008.
63. P. Korošec, J. Šilc, and B. Filipič The differential ant-stigmergy algorithm Inform. Sci. 192 1 2012 82 96
64. T.R. Levine, R. Weber, C. Hullett, H.S. Park, and L.L.M. Lindsey A critical assessment of null hypothesis significance testing in quantitative communication research Hum. Commun. Res. 34 2 2008 171 187
65. J. Li A two-step rejection procedure for testing multiple hypotheses J. Statist. Plann. Infer. 138 2008 1521 1527
66. D.V. Lindley New Cambridge Statistical Tables 1995 Cambridge University Press
67. S.-H. Liu, M. Mernik, D. Hrnčič, and M. Črepinšek A parameter control method of evolutionary algorithms using exploration and exploitation measures with a practical application for fitting Sovova's mass transfer model Appl. Soft Comput. 13 9 2013 3792 3805
68. P.B. Nemenyi, Distribution-Free Multiple Comparisons, Ph.D. thesis, Princeton University, 1963.
69. F. Neri, E. Mininno, and G. Iacca Compact particle swarm optimization Inform. Sci. 239 2013 96 121
70. F. Neri, and V. Tirronen Recent advances in differential evolution: a survey and experimental analysis R Artif. Intell. Rev. 33 1-2 2010 61 106
71. J. Neyman, and E. Pearson On the problem of the most efficient test of statistical hypothesis Philos. Trans. R. Soc. Lond. - Ser. A 231 1933 289 337
72. B. Pandolfini, Weapons of Chess: An Omnibus of Chess Strategies, Touchstone, 1989.
73. F. Pukelsheim The three sigma rule Am. Statist. 48 2 1994 88 91
74. R.V. Rao, V.J. Savsani, and D.P. Vakharia Teaching-learning-based optimization: an optimization method for continuous non-linear large scale problems Inform. Sci. 183 1 2012 1 15
75. L.A. Rastrigin The convergence of the random search method in the extremal control of a many-parameter system Autom. Rem. Control 24 10 1963 1337 1342
76. I. Rechenberg Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution Frommann-Holzboog 1973
77. D.M. Rom A sequentially rejective test procedure based on a modified Bonferroni inequality Biometrika 77 1990 663 665
78. D. Quade Using weighted rankings in the analysis of complete blocks with additive block effects J. Am. Statist. Assoc. 74 367 1979 680 683
79. N. Schenker, and J.F. Gentleman On judging the significance of differences by examining the overlap between confidence intervals Am. Statist. 55 3 2001 182 186
80. F.L. Schmid Statistical significance testing and cumulative knowledge in psychology Psychol. Meth. 1 1996 115 129
81. J.P. Shaffer Modified sequentially rejective multiple test procedures J. Am. Statist. Assoc. 81 395 1986 826 831
82. D. Sheskin Handbook of Parametric and Nonparametric Statistical Procedures 2006 Chapman & HallCRC
83. D. Shilane, J. Martikainen, S. Dudoit, and S. Ovaska A general framework for statistical performance comparisons of evolutionary computation algorithms Inform. Sci. 178 14 2008 2870 2879
84. R.G. Steel A multiple comparison sign test: treatments versus control J. Am. Statist. Assoc. 54 1959 767 775
85. R. Storn, and K. Price Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces J. Global Optim. 11 4 1997 341 359
86. Student, The probable error of a mean, Biometrika (1908) 1-25.
87. K. Tang, X. Li, P.N. Suganthan, Z. Yang, T. Weise. Benchmark Functions for the CEC'2010 Special Session and Competition on Large-Scale Global Optimization, Nature Inspired Computation and Applications Laboratory, 2009.
88. The CMA-ES Source Code < https://www.lri.fr/∼hansen/cmaes- inmatlab.html >, 2013.
89. J.W. Tukey Comparing individual means in the analysis of variance Biometrics 5 1949 99 114
90. J. Tvrdík, Adaptive differential evolution: application to nonlinear regression, in: Proceedings of the International Multiconference on Computer Science and Information Technology, 2007, pp. 193-202.
91. F. Wilcoxon Individual comparisons by ranking methods Biometrics 1 6 1945 80 83