The irace package: Iterated racing for automatic algorithm configuration

Operations Research Perspectives

Volumn 3, Issue , 2016, Pages 43-58

  López Ibáñez, Manuel ^a   Dubois Lacoste, Jérémie ^b   Pérez Cáceres, Leslie ^b   Birattari, Mauro ^b   Stützle, Thomas ^b  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

^b UNIVERSITÉ LIBRE DE BRUXELLES   (Belgium)

Author keywords

Automatic algorithm configuration; Parameter tuning; Racing

Indexed keywords

EID: 84989332245     PISSN: 22147160     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.orp.2016.09.002     Document Type: Article

References (96)

1. [1] Acosta-Mesa, H.-G., Rechy-Ramírez, F., Mezura-Montes, E., Cruz-Ramírez, N., Jiménez, R.H., Application of time series discretization using evolutionary programming for classification of precancerous cervical lesions. J Biomed Inform 49 (2014), 73–83.
2. [2] Adenso-Díaz, B., Laguna, M., Fine-tuning of algorithms using fractional experimental design and local search. Oper Res 54:1 (2006), 99–114.
3. [3] Ansótegui, C., Sellmann, M., Tierney, K., A gender-based genetic algorithm for the automatic configuration of algorithms. Gent, I.P., (eds.) Principles and practice of constraint programming, CP 2009, volume 5732 of lecture notes in computer science, 2009, Springer, Heidelberg, Germany, 142–157.
4. [4] Ansótegui, C., Malitsky, Y., Samulowitz, H., Sellmann, M., Tierney, K., Model-based genetic algorithms for algorithm configuration. Yang, Q., Wooldridge, M., (eds.) Proceedings of the twenty-fourth international joint conference on artificial intelligence (IJCAI-15), 2015, IJCAI/AAAI Press, Menlo Park, CA, 733–739.
5. [5] Audet, C., Orban, D., Finding optimal algorithmic parameters using derivative-free optimization. SIAM J Optim 17:3 (2006), 642–664.
6. [6] Audet, C., Dang, C.-K., Orban, D., Algorithmic parameter optimization of the DFO method with the OPAL framework. Naono, K., Teranishi, K., Cavazos, J., Suda, R., (eds.) Software automatic tuning: from concepts to state-of-the-art results, 2010, Springer, 255–274.
7. [7] Aydın, D., Composite artificial bee colony algorithms: from component-based analysis to high-performing algorithms. Appl Soft Comput 32 (2015), 266–285.
8. [8] Babić, D., Hu, A.J., Structural abstraction of software verification conditions. Computer aided verification: 19th international conference, CAV 2007, 2007, 366–378.
9. [9] Babić, D., Hutter, F., Spear theorem prover. SAT’08: proceedings of the SAT 2008 race, 2008.
10. [10] Balaprakash, P., Birattari, M., Stützle, T., Improvement strategies for the F-race algorithm: sampling design and iterative refinement. Bartz-Beielstein, T., Blesa, M.J., Blum, C., Naujoks, B., Roli, A., Rudolph, G., Sampels, M., (eds.) Hybrid metaheuristics, volume 4771 of lecture notes in computer science, 2007, Springer, Heidelberg, Germany, 108–122.
11. [11] Bartz-Beielstein, T., Experimental research in evolutionary computation: the new experimentalism. 2006, Springer, Berlin, Germany.
12. [12] Bartz-Beielstein, T., Lasarczyk, C., Preuss, M., Sequential parameter optimization. Proceedings of the 2005 congress on evolutionary computation (CEC 2005), 2005, IEEE Press, Piscataway, NJ, 773–780.
13. [13] Battiti, R., Brunato, M., Mascia, F., Reactive search and intelligent optimization, volume 45 of operations research/computer science interfaces. 2008, Springer, New York, NY.
14. [14] Benavides, A.J., Ritt, M., Iterated local search heuristics for minimizing total completion time in permutation and non-permutation flow shops. Brafman, R.I., Domshlak, C., Haslum, P., Zilberstein, S., (eds.) Proceedings of the twenty-fifth international conference on automated planning and scheduling, ICAPS, Jerusalem, Israel, June 7–11, 2015, AAAI Press, Menlo Park, CA, 34–41.
15. [15] Bezerra, L.C.T., López-Ibáñez, M., Stützle, T., Deconstructing multi-objective evolutionary algorithms: an iterative analysis on the permutation flowshop. Pardalos, P.M., Resende, M.G.C., Vogiatzis, C., Walteros, J.L., (eds.) Learning and intelligent optimization, 8th international conference, LION 8, volume 8426 of lecture notes in computer science, 2014, Springer, Heidelberg, Germany, 57–172.
16. [16] Bezerra, L.C.T., López-Ibáñez, M., Stützle, T., Automatic design of evolutionary algorithms for multi-objective combinatorial optimization. Bartz-Beielstein, T., Branke, J., Filipič, B., Smith, J., (eds.) PPSN 2014, volume 8672 of lecture notes in computer science, 2014, Springer, Heidelberg, Germany, 508–517.
17. [17] Bezerra, L.C.T., López-Ibáñez, M., Stützle, T., Automatic component-wise design of multi-objective evolutionary algorithms. IEEE Trans Evol Comput 20:3 (2016), 403–417.
18. [18] Birattari, M., The race package for R: racing methods for the selection of the best. Technical Report TR/IRIDIA/2003-037, 2003, IRIDIA, Université Libre de Bruxelles, Belgium.
19. [19] Birattari, M., Tuning metaheuristics: a machine learning perspective, volume 197 of studies in computational intelligence. 2009, Springer, Berlin/Heidelberg, Germany.
20. [20] Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K., A racing algorithm for configuring metaheuristics. Langdon, W.B., (eds.) Proceedings of the genetic and evolutionary computation conference, GECCO 2002, 2002, Morgan Kaufmann Publishers, San Francisco, CA, 11–18.
21. [21] Birattari, M., Yuan, Z., Balaprakash, P., Stützle, T., F-race and iterated F-race: an overview. Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M., (eds.) Experimental methods for the analysis of optimization algorithms, 2010, Springer, Berlin, Germany, 311–336.
22. [22] Bischl B., Lang M., Bossek J., Judt L., Richter J., Kuehn T., et al. mlr: machine learning in R. 2013. http://cran.r-project.org/package=mlr.R package.
23. [23] Blum, C., Calvo, B., Blesa, M.J., FrogCOL and frogMIS: new decentralized algorithms for finding large independent sets in graphs. Swarm Intell 9:2–3 (2015), 205–227.
24. [24] Blum, C., Pinacho, P., López-Ibáñez, M., Lozano, J.A., Construct, merge, solve & adapt: a new general algorithm for combinatorial optimization. Comput Oper Res 68 (2016), 75–88.
25. [25] Ceschia, S., Schaerf, A., Stützle, T., Local search techniques for a routing-packing problem. Comput Ind Eng 66:4 (2013), 1138–1149.
26. [26] Chen, Y., Hao, J.-K., Glover, F., A hybrid metaheuristic approach for the capacitated arc routing problem. Eur J Oper Res 553:1 (2016), 25–39.
27. [27] Chivilikhin, D.S., Ulyantsev, V.I., Shalyto, A.A., Modified ant colony algorithm for constructing finite state machines from execution scenarios and temporal formulas. Autom Remote Control 77:3 (2016), 473–484.
28. [28] Conover, W.J., Practical nonparametric statistics, third edition. 1999, John Wiley & Sons, New York, NY.
29. [29] Coy, S.P., Golden, B.L., Runger, G.C., Wasil, E.A., Using experimental design to find effective parameter settings for heuristics. J Heuristics 7:1 (2001), 77–97.
30. [30] Dell'Amico, M., Iori, M., Novellani, S., Stützle, T., A destroy and repair algorithm for the bike sharing rebalancing problem. Comput Oper Res 71 (2016), 146–162.
31. [31] Dubois-Lacoste, J., López-Ibáñez, M., Stützle, T., Automatic configuration of state-of-the-art multi-objective optimizers using the TP+PLS framework. Krasnogor, N., Lanzi, P.L., (eds.) Proceedings of the genetic and evolutionary computation conference, GECCO 2011, 2011, ACM Press, New York, NY, 2019–2026.
32. [32] Fawcett, C., Hoos, H.H., Analysing differences between algorithm configurations through ablation. Proceedings of MIC 2013, the 10th metaheuristics international conference, 2013, 123–132.
33. : a framework for multi-objective clustering. Haenens, C., Jourdan, L., Marmion, M.-E., (eds.) Learning and intelligent optimization, 9th international conference, LION 9, volume 8994 of lecture notes in computer science, 2015, Springer, Heidelberg, Germany, 293–305.
34. [34] Francesca, G., Brambilla, M., Brutschy, A., Trianni, V., Birattari, M., AutoMoDe: a novel approach to the automatic design of control software for robot swarms. Swarm Intell 8:2 (2014), 89–112.
35. [35] Francesca, G., Brambilla, M., Brutschy, A., Garattoni, L., Miletitch, R., Podevijn, G., et al. AutoMoDe-chocolate: automatic design of control software for robot swarms. Swarm Intell, 2015.
36. [36] Goldberg, D.E., Genetic algorithms in search, optimization and machine learning. 1989, Addison-Wesley, Boston, MA, USA.
37. [37] Grefenstette, J.J., Optimization of control parameters for genetic algorithms. IEEE Trans Syst Man Cybern 16:1 (1986), 122–128.
38. [38] Hansen, N., Ostermeier, A., Completely derandomized self-adaptation in evolution strategies. Evol Comput 9:2 (2001), 159–195.
39. [39] Herrera F., Lozano M., Molina D. Test suite for the special issue of soft computing on scalability of evolutionary algorithms and other metaheuristics for large scale continuous optimization problems. http://sci2s.ugr.es/eamhco/. 2010.
40. [40] Hoos, H.H., Programming by optimization. Commun ACM 55:2 (2012), 70–80.
41. [41] Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T., ParamILS: an automatic algorithm configuration framework. J Artif Intell Res 36 (2009), 267–306.
42. [42] Hutter, F., Hoos, H.H., Leyton-Brown, K., Automated configuration of mixed integer programming solvers. Lodi, A., Milano, M., Toth, P., (eds.) Integration of AI and OR techniques in constraint programming for combinatorial optimization problems, 7th international conference, CPAIOR 2010, volume 6140 of lecture notes in computer science, 2010, Springer, Heidelberg, Germany, 186–202.
43. [43] Hutter, F., Hoos, H.H., Leyton-Brown, K., Sequential model-based optimization for general algorithm configuration. Coello Coello, C.A., (eds.) Learning and intelligent optimization, 5th international conference, LION 5, volume 6683 of lecture notes in computer science, 2011, Springer, Heidelberg, Germany, 507–523.
44. [44] Hutter, F., López-Ibáñez, M., Fawcett, C., Lindauer, M.T., Hoos, H.H., Leyton-Brown, K., et al. AClib: a benchmark library for algorithm configuration. Pardalos, P.M., Resende, M.G.C., Vogiatzis, C., Walteros, J.L., (eds.) Learning and intelligent optimization, 8th international conference, LION 8, volume 8426 of lecture notes in computer science, 2014, Springer, Heidelberg, Germany, 36–40.
45. [45] IBM. ILOG CPLEX optimizer. http://www.ibm.com/software/integration/optimization/cplex-optimizer/.
46. [46] Jackson, C.H., Multi-state models for panel data: the msm package for R. J Stat Softw 38:8 (2011), 1–29 http://www.jstatsoft.org/v38/i08/.
47. [47] Jacquin, S., Jourdan, L., Talbi, E.-G., Dynamic programming based metaheuristic for energy planning problems. Esparcia-Alcázar, A.I., Mora, A.M., (eds.) Applications of evolutionary computation, volume 8602 of lecture notes in computer science, 2014, Springer, Heidelberg, Germany, 165–176.
48. [48] Johnson, D.S., McGeoch, L.A., The traveling salesman problem: a case study in local optimization. Aarts, E.H.L., Lenstra, J.K., (eds.) Local search in combinatorial optimization, 1997, John Wiley & Sons, Chichester, UK, 215–310.
49. [49] Johnson, D.S., McGeoch, L.A., Experimental analysis of heuristics for the STSP. Gutin, G., Punnen, A., (eds.) The traveling salesman problem and its variations, 2002, Kluwer Academic Publishers, Dordrecht, The Netherlands, 369–443.
50. [50] Karafotias, G., Hoogendoorn, M., Eiben, A.E., Parameter control in evolutionary algorithms: trends and challenges. IEEE Trans Evol Comput 19:2 (2015), 167–187.
51. [51] KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K., SATenstein: automatically building local search SAT solvers from components. Boutilier, C., (eds.) Proceedings of the twenty-first international joint conference on artificial intelligence (IJCAI-09), 2009, AAAI Press, Menlo Park, CA, 517–524.
52. [52] Lacroix, B., Molina, D., Herrera, F., Dynamically updated region based memetic algorithm for the 2013 CEC special session and competition on real parameter single objective optimization. Proceedings of the 2013 congress on evolutionary computation (CEC 2013), 2013, IEEE Press, Piscataway, NJ, 1945–1951.
53. [53] Lacroix, B., Molina, D., Herrera, F., Region based memetic algorithm for real-parameter optimisation. Inf Sci 262 (2014), 15–31.
54. [54] Lang, M., Kotthaus, H., Marwedel, Weihs, C., Rahnenführer, J., Bischl, B., Automatic model selection for high-dimensional survival analysis. J Stat Comput Simul 85:1 (2014), 62–76.
55. [55] Liao, T., Stützle, T., Benchmark results for a simple hybrid algorithm on the CEC 2013 benchmark set for real-parameter optimization. Proceedings of the 2013 congress on evolutionary computation (CEC 2013), 2013, IEEE Press, Piscataway, NJ, 1938–1944.
56. [56] Liao, T., Montes de Oca, M.A., Stützle, T., Computational results for an automatically tuned CMA-ES with increasing population size on the CEC’05 benchmark set. Soft Comput 17:6 (2013), 1031–1046.
57. [57] Liao, T., Molina, D., Stützle, T., Performance evaluation of automatically tuned continuous optimizers on different benchmark sets. Appl Soft Comput 27 (2015), 490–503.
58. [58] López-Ibáñez, M., Stützle, T., The automatic design of multi-objective ant colony optimization algorithms. IEEE Trans Evol Comput 16:6 (2012), 861–875.
59. [59] López-Ibáñez, M., Stützle, T., Automatically improving the anytime behaviour of optimisation algorithms. Eur J Oper Res 235:3 (2014), 569–582.
60. [60] López-Ibáñez, M., Blum, C., Ohlmann, J.W., Thomas, B.W., The travelling salesman problem with time windows: adapting algorithms from travel-time to makespan optimization. Appl Soft Comput 13:9 (2013), 3806–3815.
61. [61] López-Ibáñez M., Dubois-Lacoste J., Pérez Cáceres L., Stützle T., Birattari M. 2016a. The irace package: Iterated racing for automatic algorithm configuration. http://iridia.ulb.ac.be/supp/IridiaSupp2016-003/.
62. [62] López-Ibáñez, M., Pérez Cáceres, L., Dubois-Lacoste, J., Stützle, T., Birattari, M., The irace package: user guide. Technical Report TR/IRIDIA/2016-004, 2016, IRIDIA, Université Libre de Bruxelles, Belgium http://iridia.ulb.ac.be/IridiaTrSeries/IridiaTr2016-004.pdf.
63. [63] Marmion, M.-E., Mascia, F., López-Ibáñez, M., Stützle, T., Automatic design of hybrid stochastic local search algorithms. Blesa, M.J., Blum, C., Festa, P., Roli, A., Sampels, M., (eds.) Hybrid metaheuristics, volume 7919 of lecture notes in computer science, 2013, Springer, Heidelberg, Germany, 144–158.
64. [64] Maron, O., Moore, A.W., The racing algorithm: model selection for lazy learners. Artif Intell Res 11:1–5 (1997), 193–225.
65. [65] Mascia, F., Birattari, M., Stützle, T., Tuning algorithms for tackling large instances: an experimental protocol. Pardalos, P., Nicosia, G., (eds.) Learning and intelligent optimization, 7th international conference, LION 7, volume 7997 of lecture notes in computer science, 2013, Springer, Heidelberg, Germany, 410–422.
66. [66] Mascia, F., López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T., Grammar-based generation of stochastic local search heuristics through automatic algorithm configuration tools. Comput Oper Res 51 (2014), 190–199.
67. [67] Massen, F., López-Ibáñez, M., Stützle, T., Deville, Y., Experimental analysis of pheromone-based heuristic column generation using irace. Blesa, M.J., Blum, C., Festa, P., Roli, A., Sampels, M., (eds.) Hybrid metaheuristics, volume 7919 of lecture notes in computer science, 2013, Springer, Heidelberg, Germany, 92–106.
68. [68] Meier J.F., Clausen U. 2014. A versatile heuristic approach for generalized hub location problems. Preprint, Provided upon personal request.
69. [69] Mesquita, R.G., Silva, R.M., Mello, C.A., Miranda, P.B., Parameter tuning for document image binarization using a racing algorithm. Expert Syst Appl 42:5 (2015), 2593–2603.
70. [70] Miranda, P., Silva, R.M., Prudêncio, R.B., Fine-tuning of support vector machine parameters using racing algorithms. 22st European symposium on artificial neural networks, computational intelligence And machine learning, Bruges, 2014, ESANN, 325–330.
71. [71] Montes de Oca, M.A., Aydın, D., Stützle, T., An incremental particle swarm for large-scale continuous optimization problems: an example of tuning-in-the-loop (re)design of optimization algorithms. Soft Comput 15:11 (2011), 2233–2255.
72. [72] Mühlenthaler, M., Fairness in academic course timetabling. 2015, Springer.
73. [73] Nannen, V., Eiben, A.E., A method for parameter calibration and relevance estimation in evolutionary algorithms. Cattolico, M., (eds.) Proceedings of the genetic and evolutionary computation conference, GECCO 2006, 2006, ACM Press, New York, NY, 183–190.
74. [74] Nannen, V., Eiben, A.E., Relevance estimation and value calibration of evolutionary algorithm parameters. Veloso, M.M., (eds.) Proceedings of the twentieth international joint conference on artificial intelligence (IJCAI-07), 2007, AAAI Press, Menlo Park, CA, 975–980.
75. [75] Nashed, Y.S.G., Mesejo, P., Ugolotti, R., Dubois-Lacoste, J., Cagnoni, S., A comparative study of three GPU-based metaheuristics. Coello Coello, C.A., (eds.) PPSN 2012, part II, volume 7492 of lecture notes in computer science, 2012, Springer, Heidelberg, Germany, 398–407.
76. [76] Pellegrini, P., Birattari, M., Stützle, T., A critical analysis of parameter adaptation in ant colony optimization. Swarm Intell 6:1 (2012), 23–48.
77. [77] Pellegrini, P., Castelli, L., Pesenti, R., Metaheuristic algorithms for the simultaneous slot allocation problem. IET Intell Transport Syst 6:4 (2012), 453–462.
78. [78] Pérez Cáceres, L., López-Ibáñez, M., Stützle, T., An analysis of parameters of irace. Blum, C., Ochoa, G., (eds.) Proceedings of EvoCOP 2014–14th European conference on evolutionary computation in combinatorial optimization, volume 8600 of lecture notes in computer science, 2014, Springer, Heidelberg, Germany, 37–48.
79. [79] Powell, M., The BOBYQA algorithm for bound constrained optimization without derivatives. Technical Report Cambridge NA Report NA2009/06, 2009, University of Cambridge, UK.
80. [80] Ridge, E., Kudenko, D., Tuning the performance of the MMAS heuristic. Stützle, T., Birattari, M., Hoos, H.H., (eds.) International workshop on engineering stochastic local search algorithms (SLS 2007), volume 4638 of lecture notes in computer science, 2007, Springer, Heidelberg, Germany, 46–60.
81. [81] Riff, M.-C., Montero, E., A new algorithm for reducing metaheuristic design effort. Proceedings of the 2013 congress on evolutionary computation (CEC 2013), 2013, IEEE Press, Piscataway, NJ, 3283–3290.
82. [82] Robert, C.P., Simulation of truncated normal variables. Stat Comput 5:2 (1995), 121–125.
83. [83] Ruiz, R., Maroto, C., A comprehensive review and evaluation of permutation flow-shop heuristics. Eur J Oper Res 165:2 (2005), 479–494.
84. [84] Samà, M., Pellegrini, P., Ariano, A.D., Rodriguez, J., Pacciarelli, D., Ant colony optimization for the real-time train routing selection problem. Transp Res Part B 85 (2016), 89–108.
85. [85] Schneider, M., Hoos, H.H., Quantifying homogeneity of instance sets for algorithm configuration. Hamadi, Y., Schoenauer, M., (eds.) Learning and intelligent optimization, 6th international conference, LION 6, volume 7219 of lecture notes in computer science, 2012, Springer, Heidelberg, Germany, 190–204.
86. [86] Stefanello, F., Aggarwal, V., Buriol, L.S., Gonçalves, J.F., Resende, M.G.C., A biased random-key genetic algorithm for placement of virtual machines across geo-separated data centers. Silva, S., Esparcia-Alcázar, A.I., (eds.) Proceedings of the genetic and evolutionary computation conference, GECCO 2015, 2015, ACM Press, New York, NY, 919–926.
87. [87] Stützle T.ACOTSP: a software package of various ant colony optimization algorithms applied to the symmetric traveling salesman problem. 2002. http://www.aco-metaheuristic.org/aco-code/.
88. [88] Styles, J., Hoos, H.H., Ordered racing protocols for automatically configuring algorithms for scaling performance. Blum, C., Alba, E., (eds.) Proceedings of the genetic and evolutionary computation conference, GECCO 2013, 2013, ACM Press, New York, NY, 551–558.
89. [89] Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K., Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. Dhillon, I.S., Koren, Y., Ghani, R., Senator, T.E., Bradley, P., Parekh, R., He, J., Grossman, R.L., Uthurusamy, R., (eds.) The 19th ACM SIGKDD international conference on knowledge discovery and data mining, KDD 2013, 2013, ACM Press, New York, NY, 847–855.
90. [90] Violin, A., Mathematical programming approaches to pricing problems, 2014, Faculté de Sciences, Université Libre de Bruxelles and Dipartimento di Ingegneria e Architettura, Università degli studi di Trieste Phd thesis.
91. [91] Wessing, S., Beume, N., Rudolph, G., Naujoks, B., Parameter tuning boosts performance of variation operators in multiobjective optimization. Schaefer, R., Cotta, C., Kolodziej, J., Rudolph, G., (eds.) Parallel problem solving from nature, PPSN XI, volume 6238 of lecture notes in computer science, 2010, Springer, Heidelberg, Germany, 728–737.
92. [92] Yarimcam, A., Asta, S., Ozcan, E., Parkes, A.J., Heuristic generation via parameter tuning for online bin packing. Evolving and autonomous learning systems (EALS), 2014 IEEE symposium on, 2014, IEEE, 102–108.
93. [93] Yuan, Z., Montes de Oca, M.A., Stützle, T., Birattari, M., Continuous optimization algorithms for tuning real and integer algorithm parameters of swarm intelligence algorithms. Swarm Intell 6:1 (2012), 49–75.
94. [94] Yuan, Z., Montes de Oca, M.A., Stützle, T., Lau, H.C., Birattari, M., An analysis of post-selection in automatic configuration. Blum, C., Alba, E., (eds.) Proceedings of GECCO 2013, 2013, ACM Press, New York, NY, 1557–1564.
95. [95] Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C.M., Fonseca, V.G.d., Performance assessment of multiobjective optimizers: an analysis and review. IEEE Trans Evol Comput 7:2 (2003), 117–132.
96. [96] Zlochin, M., Birattari, M., Meuleau, N., Dorigo, M., Model-based search for combinatorial optimization: acritical survey. Ann Oper Res 131:1–4 (2004), 373–395.