Tradeoffs in the empirical evaluation of competing algorithm designs

Annals of Mathematics and Artificial Intelligence

Volumn 60, Issue 1, 2010, Pages 65-89

  Hutter, Frank ^a   Hoos, Holger H ^a   Leyton Brown, Kevin ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Algorithm configuration; Algorithm design; Empirical analysis; Parameter optimization

Indexed keywords

EID: 79959376680     PISSN: 10122443     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10472-010-9191-0     Document Type: Article

References (35)

1. Adenso-Diaz, B., Laguna, M.: Fine-tuning of algorithms using fractional experimental design and local search. Oper. Res. 54(1), 99-114 (2006).
2. Ansotegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of solvers. In: Proc. of CP-09, pp. 142-157 (2009).
3. Audet, C., Orban, D.: Finding optimal algorithmic parameters using the mesh adaptive direct search algorithm. SIAM J. Optim. 17(3), 642-664 (2006).
4. Babić, D., Hu, A. J.: Structural abstraction of software verification conditions. In: Proc. of CAV-07, pp. 366-378 (2007).
5. Balaprakash, P., Birattari, M., Stützle, T.: Improvement strategies for the F-Race algorithm: sampling design and iterative refinement. In: Proc. of MH'07, pp. 108-122 (2007).
6. Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation: the New Experimentalism. Natural Computing Series. Springer Verlag, Berlin (2006).
7. Beasley, J.: OR-Library: distributing test problems by electronic mail. J. Oper. Res. Soc. 41(11), 1069-1072 (1990).
8. Birattari, M.: The Problem of Tuning Metaheuristics as Seen from a Machine Learning Perspective. DISKI 292, Infix/Aka, Berlin, Germany (2005).
9. Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proc. of GECCO-02, pp. 11-18 (2002).
10. Etzioni, O., Etzioni, R.: Statistical methods for analyzing speedup learning experiments. J. Mach. Learn. Res. 14(3), 333-347 (1994).
11. Gent, I. P., Hoos, H. H., Prosser, P., Walsh, T.: Morphing: combining structure and randomness. In: Proc. of AAAI-99, pp. 654-660 (1999).
12. Gomes, C. P., Selman, B.: Problem structure in the presence of perturbations. In: Proc. of AAAI-97, pp. 221-226 (1997).
13. Gomes, C. P., Selman, B.: Algorithm portfolios. Artif. Intell. 126(1-2), 43-62 (2001).
14. Gratch, J., Dejong, G.: Composer: A probabilistic solution to the utility problem in speed-up learning. In: Proc. of AAAI-92, pp. 235-240 (1992).
15. Hastie, T., Tibshirani, R., Friedman, J. H.: The Elements of Statistical Learning. Springer Series in Statistics. Springer Verlag (2001).
16. Hoos, H. H.: Computer-aided Design of High-performance Algorithms. Technical Report TR-2008-16, University of British Columbia, Department of Computer Science (2008).
17. Horvitz, E., Ruan, Y., Gomes, C. P., Kautz, H., Selman, B., Chickering, D. M.: A Bayesian approach to tackling hard computational problems. In: Proc. of UAI-01, pp. 235-244 (2001).
18. Hutter, F.: Automated configuration of algorithms for solving hard computational problems. Ph. D. thesis, University of British Columbia, Department of Computer Science, Vancouver, Canada (2009).
19. Hutter, F., Babić, D., Hoos, H. H., Hu, A. J.: Boosting verification by automatic tuning of decision procedures. In: Proc. of FMCAD-07, pp. 27-34 (2007).
20. Hutter, F., Hamadi, Y., Hoos, H. H., Leyton-Brown, K.: Performance prediction and automated tuning of randomized and parametric algorithms. In: Proc. of CP-06, pp. 213-228 (2006).
21. Hutter, F., Hoos, H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. JAIR 36, 267-306 (2009).
22. Hutter, F., Hoos, H. H., Leyton-Brown, K., Murphy, K. P.: An experimental investigation of model-based parameter optimisation: SPO and beyond. In: Proc. of GECCO-09, pp. 271-278 (2009).
23. Hutter, F., Hoos, H. H., Leyton-Brown, K., Murphy, K. P.: Time-bounded sequential parameter optimization. In: Proc. of LION-10 (2010, to appear).
24. Hutter, F., Hoos, H. H., Stützle, T.: Automatic algorithm configuration based on local search. In: Proc. of AAAI-07, pp. 1152-1157 (2007).
25. KhudaBukhsh, A. R., Xu, L., Hoos, H. H., Leyton-Brown, K.: Satenstein: automatically building local search SAT solvers from components. In: Proc. of IJCAI-09, pp. 517-524 (2009).
26. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Learning the empirical hardness of optimization problems: the case of combinatorial auctions. In: Proc. of CP-02, pp. 556-572 (2002).
27. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Empirical hardness models: methodology and a case study on combinatorial auctions. J. ACM 56(4), 1-52 (2009).
28. Leyton-Brown, K., Pearson, M., Shoham, Y.: Towards a universal test suite for combinatorial auction algorithms. In: Proc. of EC-00, pp. 66-76 (2000).
29. Maron, O., Moore, A.: Hoeffding races: Accelerating model selection search for classification and function approximation. In: Proc. of NIPS-94, pp. 59-66 (1994).
30. Minton, S.: Automatically configuring constraint satisfaction programs: a case study. Constraints 1(1), 1-40 (1996).
31. Nudelman, E., Leyton-Brown, K., Hoos, H. H., Devkar, A., Shoham, Y.: Understanding random SAT: beyond the clauses-to-variables ratio. In: Proc. of CP-04 (2004).
32. Segre, A., Elkan, C., Russell, A.: A critical look at experimental evaluations of EBL. J. Mach. Learn. 6(2), 183-195 (1991).
33. Simon, L., Chatalic, P.: SATEx: a web-based framework for SAT experimentation. In: Proc. of SAT-01 (2001).
34. Xu, L., Hutter, F., Hoos, H. H., Leyton-Brown, K.: SATzilla: portfolio-based algorithm selection for SAT. J. Artif. Intell. Res. 32, 565-606 (2008).
35. Zarpas, E.: Benchmarking SAT solvers for bounded model checking. In: Proc. of SAT-05, pp. 340-354 (2005).