A multicriteria statistical based comparison methodology for evaluating evolutionary algorithms

IEEE Transactions on Evolutionary Computation

Volumn 15, Issue 6, 2011, Pages 848-870

  Carrano, Eduardo G ^a   Wanner, Elizabeth F ^a   Takahashi, Ricardo H C ^b  

^a Centro Federal de Educação Tecnol gica de Minas Gerais   (Brazil)

^b FEDERAL UNIVERSITY OF MINAS GERAIS   (Brazil)

Author keywords

Algorithm evaluation; evolutionary algorithms; multicriteria statistical comparison

Indexed keywords

ALGORITHM EVALUATION; ARBITRARY PROBABILITY DISTRIBUTION; BENCH-MARK PROBLEMS; CORRELATION EFFECT; EFFICIENT SETS; FIRST ORDER; MULTI CRITERIA ANALYSIS; MULTI-CRITERIA; MULTICRITERIA STATISTICAL COMPARISON; NON-GAUSSIAN; P-VALUES; PERMUTATION TESTS; PRE-PROCESSING; REFERENCE PROBABILITY; STOCHASTIC DOMINANCE;

BENCHMARKING; DISTRIBUTION FUNCTIONS; FUNCTION EVALUATION; PRINCIPAL COMPONENT ANALYSIS; PROBABILITY DENSITY FUNCTION; PROBABILITY DISTRIBUTIONS;

EVOLUTIONARY ALGORITHMS;

EID: 82455172021     PISSN: 1089778X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TEVC.2010.2069567     Document Type: Article

References (40)

1. P. Dutta and D. Dutta Majumder, "Performance comparison of two evolutionary schemes," in Proc. Int. Conf. Pattern Recognit., 1996, pp. 659-663.
2. B. G. W. Craenen, A. E. Eiben, and J. I. van Hemert, "Comparing evolutionary algorithms on binary constraint satisfaction problems," IEEE Trans. Evol. Comput., vol. 7, no. 5, pp. 424-444, Oct. 2003.
3. R. H. C. Takahashi, J. A. Vasconcelos, J. A. Ramirez, and L. Krahenbuhl, "A multiobjective methodology for evaluating genetic operators," IEEE Trans. Magn., vol. 39, no. 3, pp. 1321-1324, May 2003.
4. E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, and V. G. Fonseca, "Performance assessment of multiobjective optimizers: An analysis and review," IEEE Trans. Evol. Comput., vol. 7, no. 2, pp. 117-132, Apr. 2003.
5. D. Shilane, J. Martikainen, S. Dudoit, and S. Ovaska, "A general framework for statistical performance comparison of evolutionary computation algorithms," in Proc. Artif. Intell. Applicat. Conf., 2006, pp. 7-12.
6. E. G. Carrano, C. M. Fonseca, R. H. C. Takahashi, L. C. A. Pimenta, and O. M. Neto, "A preliminary comparison of tree encoding schemes for evolutionary algorithms," in Proc. IEEE Int. Conf. Syst. Man Cybern., Oct. 2007, pp. 1969-1974.
7. S. Garcia, D. Molina, M. Lozano, and F. Herrera, "A study on the use of non-parametric tests for analyzing the evolutionary algorithms' behavior: A case study on the CEC'2005 special session on real parameter optimization," J. Heuristics, vol. 15, no. 6, pp. 617-644, 2008.
8. M. Ehrgott, Multicriteria Optimization. Berlin, Germany: Springer, 2000.
9. A. Czarn, C. MacNish, K. Vijayan, B. Turlach, and R. Gupta, "Statistical exploratory analysis of genetic algorithms," IEEE Trans. Evol. Comput., vol. 8, no. 4, pp. 405-421, Aug. 2004.
10. H. R. Lindman, Analysis of Variance in Complex Experimental Designs. San Francisco, CA: Freeman, 1974.
11. B. Yuan and M. Gallagher, "Statistical racing techniques for improved empirical evaluation of evolutionary algorithms," in Proc. Parallel Problem Solving Nature, 2004, pp. 172-181.
12. W. J. Connover, Practical Nonparametric Statistics, 3rd ed. New York: Wiley, 1999.
13. H. Mühlenbein and G. Paaß, "From recombination of genes to the estimation of distributions: I. Binary parameters," in Proc. Parallel Problem Solving Nature, 1996, pp. 178-187.
14. B. Efron, "Bootstrap methods: Another look at the jackknife," Ann. Statist., vol. 7, no. 1, pp. 1-26, 1979.
15. T. C. Hu, "Optimum communication spanning trees," SIAM J. Comput., vol. 3, no. 3, pp. 188-195, 1974.
16. E. G. Carrano, R. H. C. Takahashi, and E. F. Wanner, "An enhanced statistical approach for evolutionary algorithm comparison," in Proc. Genet. Evol. Comput. Conf., 2008, pp. 897-904.
17. S. Pemmaraju and S. Skiena, Implementing Discrete Mathematics: Combinatorics and Graph Theory with Mathematica. Cambridge, MA: Cambridge University Press, 2003.
18. K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed. Amsterdam, The Netherlands: Elsevier, 1990.
19. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed. Reading, MA: Addison-Wesley, 1992.
20. M. R. Chernick, Bootstrap Methods, A Practitioner's Guide (Wiley Series in Probability and Statistics). New York: Wiley, 1999.
21. B. Efron, "Nonparametric estimates of standard error: The jackknife, the bootstrap and other methods," Biometrika, vol. 68, no. 3, pp. 589-599, 1981.
22. B. Efron, "The jackknife, the bootstrap, and other resampling plans," in Proc. CBMS-NSF Reg. Conf. Series Appl. Math., vol. 38. 1982.
23. J. Rice, Mathematical Statistics and Data Analysis, 2nd ed. Pacific Grove, CA: Duxbury Press, 1994.
24. O. Kallenberg, Foundations of Modern Probability. New York: Springer-Verlag, 1997.
25. E. G. Carrano, R. H. C. Takahashi, C. M. Fonseca, and O. M. Neto, "Nonlinear network optimization: An embedding vector space approach," IEEE Trans. Evol. Comput., vol. 14, no. 2, pp. 206-226, Apr. 2010.
26. L. R. Linton and L. D. Harder, Biology -Quantitative Biology Lecture Notes. Calgary, AL, Canada: University of Calgary, 2007.
27. G. E. P. Box, "Effects of inequality of variance and correlation between errors in the two way classification," Ann. Math. Statist., vol. 25, no. 3, pp. 484-498, 1954.
28. D. C. Montgomery, Design and Analysis of Experiments, 5th ed. New York: Wiley, 2001.
29. L. A. Franklin and B. J. Cooley. (2002, Mar.). An experiential approach to integrating ANOVA concepts. J. Statist. Edu.
30. [Online]. 10(1). Available: http://www.amstat.org/publications/jse/v10n1/ datasets. franklin.html
31. D. S. Moore and G. P. McCabe, Introduction to the Practice of Statistics, 5th ed. San Francisco, CA: Freeman, 2005.
32. E. Oja, "Neural networks, principal component, and subspaces," Int. J. Neural Syst., vol. 1, no. 1, pp. 61-68, 1989.
33. I. T. Jolliffe, Principal Component Analysis, 2nd ed. New York: Springer, 2002.
34. J. L. Hodges, Jr. and E. L. Lehmann, "The efficiency of some nonparametric competitors of the t-test," Ann. Math. Statist., vol. 27, no. 2, pp. 324-335, 1956.
35. W. E. Franck, "A likelihood ratio test for stochastic ordering," J. Am. Statist. Assoc., vol. 79, no. 387, pp. 686-691, 1984.
36. J. Capon, "On the asymptotic efficiency of the Kolmogorov-Smirnov test," J. Am. Statist. Assoc., vol. 60, no. 311, pp. 843-853, 1965.
37. J. J. Liang, T. P. Runarsson, E. Mezura-Montes, M. Clerc, P. N. Suganthan, C. A. C. Coello, and K. Deb, "Problem definitions and evaluation criteria for CEC 2006 special section on constraint real-parameter optimization," Nanyang Technol. Univ./University of Iceland/CINVESTAVIPN/ France T́eĺecom/Indian Institute of Technology, Vancouver, BC, Canada, Tech. Rep., 2006.
38. E. Mezura-Montes and B. Lopez-Ramirez, "Comparing bio-inspired algorithms in constrained optimization problems," in Proc. IEEE Congr. Evol. Comput., Vancouver, Canada, Sep. 2007, pp. 662-666.
39. K. Deb, "An efficient constraint handling method for genetic algorithms," Comput. Methods Appl. Mech. Eng., vol. 186, nos. 2-4, pp. 311-338, Jun. 2000.
40. E. Mezura-Montes and C. A. C. Coello, "A simple multimembered evolution strategy to solve constrained optimization problems," IEEE Trans. Evol. Comput., vol. 9, no. 1, pp. 1-17, Feb. 2005.