How to assess and report the performance of a stochastic algorithm on a benchmark problem: Mean or best result on a number of runs?

Optimization Letters

Volumn 1, Issue 3, 2007, Pages 309-311

  Birattari, Mauro ^a   Dorigo, Marco ^a  

^a UNIVERSITÉ LIBRE DE BRUXELLES   (Belgium)

Author keywords

Assessment of performance; Experimental methodology; Metaheuristics; Stochastic algorithms

Indexed keywords

ALGORITHMS; HEURISTIC METHODS; NUMBER THEORY; PROBLEM SOLVING; STATISTICS;

ASSESSMENT OF PERFORMANCE; EXPERIMENTAL METHODOLOGY; METAHEURISTICS; STOCHASTIC ALGORITHMS;

STOCHASTIC PROGRAMMING;

EID: 34249079723     PISSN: 18624472     EISSN: 18624480     Source Type: Journal    
DOI: 10.1007/s11590-006-0011-8     Document Type: Article

References (9)

1. Eiben, A.E., Jelasity, M.: A critical note on experimental research methodology in EC. In: Proceedings of the 2002 Congress on Evolutionary Computation (CEC'2002), pp. 582-587. IEEE, New York (2002)
2. Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Springer, Berlin Heidelberg New York (2003)
3. Fleming, P.J., Wallace, J.J.: How not to lie with statistics: the correct way to summarize benchmark results. Commun. ACM 29(3), 218-221 (1986)
4. Barr, R.S., Golden, B.L., Kelly, J.P., Resende, M.G.C., Stewart, W.R.: Designing and reporting computational experiments with heuristic methods. J. Heuristics 1(1), 9-32 (1995)
5. Hooker, J.N.: Testing heuristics: We have it all wrong. J. Heuristics 1(1), 33-42 (1995)
6. Rardin, R.R., Uzsoy, R.: Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7(2), 261-304 (2001)
7. Stützle, T.O.: Local Search Algorithms for Combinatorial Problems - Analysis, Algorithms, and New Applications. PhD Thesis, Technische Universität Darmstadt (1999)
8. Glover, F., Kochenberger, G. (eds.): Handbook of Metaheuristics. Kluwer, Norwell (2002)
9. Hoos, H.H., Stützle, T.: Stochastic Local Search. Foundations and Applications. Morgan Kaufmann, San Francisco (2004)