Algorithm runtime prediction: Methods & evaluation

Artificial Intelligence

Volumn 206, Issue 1, 2014, Pages 79-111

  Hutter, Frank ^a   Xu, Lin ^a   Hoos, Holger H ^a   Leyton Brown, Kevin ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Empirical performance models; Highly parameterized algorithms; Mixed integer programming; Performance prediction; Propositional satisfiability; Response surface models; Supervised machine learning; Travelling salesperson problem

Indexed keywords

FORECASTING; FORMAL LOGIC; GENETIC ALGORITHMS; INTEGER PROGRAMMING; LEARNING ALGORITHMS; LEARNING SYSTEMS; PARAMETERIZATION; SUPERVISED LEARNING;

EMPIRICAL PERFORMANCE; MIXED INTEGER PROGRAMMING; PARAMETERIZED ALGORITHM; PERFORMANCE PREDICTION; PROPOSITIONAL SATISFIABILITY; RESPONSE SURFACE MODELS; SUPERVISED MACHINE LEARNING; TRAVELLING SALESPERSON PROBLEMS;

PARAMETER ESTIMATION;

EID: 84887848457     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2013.10.003     Document Type: Article

References (122)

1. K. Ahmadizadeh, B. Dilkina, C. Gomes, and A. Sabharwal An empirical study of optimization for maximizing diffusion in networks Proceedings of the 16th International Conference on Principles and Practice of Constraint Programming (CPE'10) LNCS vol. 6308 2010 Springer-Verlag 514 521
2. D.L. Applegate, R.E. Bixby, V. Chvátal, and W.J. Cook The Traveling Salesman Problem: A Computational Study 2006 Princeton University Press
3. D. Babić Exploiting structure for scalable software verification PhD thesis 2008 University of British Columbia Vancouver, Canada
4. D. Babić, and A.J. Hu Structural abstraction of software verification conditions Proceedings of the 19th International Conference on Computer Aided Verification (CAVE'07) LNCS vol. 4590 2007 Springer-Verlag 366 378
5. D. Babić, and F. Hutter Spear theorem prover. Solver description SAT competition 2007 2007
6. T. Bartz-Beielstein Experimental Research in Evolutionary Computation: The New Experimentalism Natural Computing Series 2006 Springer
7. T. Bartz-Beielstein, C. Lasarczyk, and M. Preuss Sequential parameter optimization Proceedings of the 2004 Congress on Evolutionary Computation (CECE'05) 2005 773 780 (Pubitemid 41495929)
8. T. Bartz-Beielstein, and S. Markon Tuning search algorithms for real-world applications: a regression tree based approach Proceedings of the 2004 Congress on Evolutionary Computation (CECE'04) 2004 1111 1118
9. J. Bergstra, and Y. Bengio Random search for hyper-parameter optimization J. Mach. Learn. Res. 13 2012 281 305
10. M. Berkelaar, J. Dirks, K. Eikland, P. Notebaert, J. Ebert, lp-solve 5.5, http://lpsolve.sourceforge.net/5.5/index.htm, 2012. Last accessed on August 6, 2012.
11. T. Berthold, G. Gamrath, S. Heinz, M. Pfetsch, S. Vigerske, K. Wolter, SCIP 1.2.1.4, http://scip.zib.de/doc/html/index.shtml, 2012. Last accessed on August 6, 2012.
12. C.M. Bishop Pattern Recognition and Machine Learning 2006 Springer
13. G.E.P. Box, and N. Draper Response Surfaces, Mixtures, and Ridge Analyses second edition 2007 Wiley
14. G.E.P. Box, and K. Wilson On the experimental attainment of optimum conditions (with discussion) J. R. Stat. Soc. B 13 1 1951 1 45
15. L. Breiman Random forests Mach. Learn. 45 1 2001 5 32 (Pubitemid 32933532)
16. L. Breiman, J.H. Friedman, R. Olshen, and C.J. Stone Classification and Regression Trees 1984 Wadsworth
17. E.A. Brewer Portable high-performance supercomputing: high-level platform-dependent optimization PhD thesis 1994 Massachusetts Institute of Technology
18. E.A. Brewer High-level optimization via automated statistical modeling Proceedings of the 5th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPOPP-95) 1995 80 91
19. P. Cheeseman, B. Kanefsky, and W.M. Taylor Where the really hard problems are Proceedings of the 9th National Conference on Artificial Intelligence (AAAIE'91) 1991 331 337
20. M. Chiarandini, and Y. Goegebeur Mixed models for the analysis of optimization algorithms T. Bartz-Beielstein, M. Chiarandini, L. Paquete, M. Preuss, Experimental Methods for the Analysis of Optimization Algorithms 2010 Springer-Verlag 225 264
21. W. Cook Applications of the TSP http://www.tsp.gatech.edu/apps/index.html 2012 Last accessed on April 10, 2012
22. W. Cook Concorde downloads page http://www.tsp.gatech.edu/concorde/ downloads/downloads.htm 2012 Last accessed on October 24, 2012
23. N. Eén, and A. Biere Effective preprocessing in SAT through variable and clause elimination Proceedings of the 7th International Conference on Theory and Applications of Satisfiability Testing (SATE'04) LNCS vol. 3569 2005 Springer-Verlag 61 75 (Pubitemid 41425497)
24. N. Eén, and N. Sörensson An extensible SAT-solver Proceedings of the 6th International Conference on Theory and Applications of Satisfiability Testing (SATE'03) 2004 502 518
25. E. Ertin Gaussian process models for censored sensor readings Proceedings of the IEEE Statistical Signal Processing Workshop 2007 (SSPE'07) 2007 665 669
26. E. Fink How to solve it automatically: Selection among problem-solving methods Proceedings of the Fourth International Conference on AI Planning Systems 1998 AAAI Press 128 136
27. M. Gagliolo, and C. Legrand Algorithm survival analysis T. Bartz-Beielstein, M. Chiarandini, L. Paquete, M. Preuss, Experimental Methods for the Analysis of Optimization Algorithms 2010 Springer 161 184
28. M. Gagliolo, and J. Schmidhuber Dynamic algorithm portfolios International Symposium on Artificial Intelligence and Mathematics (ISAIME'06) 2006
29. C. Gebruers, A. Guerri, B. Hnich, and M. Milano Making choices using structure at the instance level within a case based reasoning framework Proceedings of the International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (CPAIORE'04) LNCS vol. 3011 2004 Springer-Verlag 380 386
30. C. Gebruers, B. Hnich, D. Bridge, and E. Freuder Using CBR to select solution strategies in constraint programming Proceedings of the 6th International Conference on Case Based Reasoning (ICCBRE'05) LNCS vol. 3620 2005 Springer-Verlag 222 236 (Pubitemid 41480030)
31. C.P. Gomes, B. Selman, N. Crato, and H. Kautz Heavy-tailed phenomena in satisfiability and constraint satisfaction problems J. Autom. Reason. 24 1 2000 67 100 (Pubitemid 30576096)
32. C.P. Gomes, W.-J. van Hoeve, and A. Sabharwal Connections in networks: a hybrid approach Proceedings of the International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (CPAIORE'08) LNCS vol. 5015 2008 Springer-Verlag 303 307 (Pubitemid 351777390)
33. A. Guerri, and M. Milano Learning techniques for automatic algorithm portfolio selection Proceedings of the 16th European Conference on Artificial Intelligence (ECAIE'04) 2004 475 479
34. H. Guo, and W.H. Hsu A learning-based algorithm selection meta-reasoner for the real-time MPE problem Proceedings of the 17th Australian Conference on Artificial Intelligence (AIE'04) LNCS vol. 3339 2004 Springer-Verlag 307 318 (Pubitemid 41050363)
35. Gurobi Optimization Inc. Gurobi 2.0. http://www.gurobi.com/ Last accessed on August 6, 2012
36. I. Guyon, S. Gunn, M. Nikravesh, and L. Zadeh Feature Extraction, Foundations and Applications 2006 Springer
37. S. Haim, and T. Walsh Online estimation of SAT solving runtime Proceedings of the 11th International Conference on Theory and Applications of Satisfiability Testing (SATE'08) LNCS vol. 4996 2008 Springer-Verlag 133 138
38. E.A. Hansen, and S. Zilberstein Monitoring the progress of anytime problem-solving Proceedings of the Thirteenth National Conference on Artificial Intelligence Portland, Oregon 1996 1229 1234
39. T. Hastie, R. Tibshirani, and J.H. Friedman The Elements of Statistical Learning second edition Springer Series in Statistics 2009 Springer
40. K. Helsgaun An effective implementation of the Lin-Kernighan traveling salesman heuristic Eur. J. Oper. Res. 126 1 2000 106 130
41. P. Herwig Using graphs to get a better insight into satisfiability problems Master's thesis 2006 Delft University of Technology, Department of Electrical Engineering, Mathematics and Computer Science
42. H.H. Hoos, and T. Stützle Stochastic Local Search - Foundations & Applications 2005 Morgan Kaufmann Publishers
43. E. Horvitz, Y. Ruan, C.P. Gomes, H. Kautz, B. Selman, and D.M. Chickering A Bayesian approach to tackling hard computational problems Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (UAIE'01) 2001 235 244
44. T. Hothorn, B. Lausen, A. Benner, and M. Radespiel-Tröger Bagging survival trees Stat. Med. 23 2004 77 91 (Pubitemid 38072108)
45. A.E. Howe, E. Dahlman, C. Hansen, M. Scheetz, and A. Mayrhauser Exploiting competitive planner performance S. Biundo, M. Fox, Recent Advances in AI Planning (ECPE'99) Lecture Notes in Computer Science vol. 1809 2000 Springer Berlin, Heidelberg 62 72
46. E.I. Hsu, C. Muise, J.C. Beck, and S.A. McIlraith Probabilistically estimating backbones and variable bias: experimental overview Proceedings of the 14th International Conference on Principles and Practice of Constraint Programming (CPE'08) LNCS vol. 5202 2008 Springer-Verlag 613 617
47. L. Huang, J. Jia, B. Yu, B. Chun, P. Maniatis, and M. Naik Predicting execution time of computer programs using sparse polynomial regression Proceedings of the 23rd Conference on Advances in Neural Information Processing Systems (NIPSE'10) 2010 883 891
48. F. Hutter Automated configuration of algorithms for solving hard computational problems PhD thesis 2009 University of British Columbia, Department of Computer Science Vancouver, Canada
49. F. Hutter, D. Babić, H.H. Hoos, and A.J. Hu Boosting verification by automatic tuning of decision procedures Proceedings of the 7th International Conference on Formal Methods in Computer-Aided Design (FMCADE'07) 2007 27 34
50. F. Hutter, Y. Hamadi, H.H. Hoos, and K. Leyton-Brown Performance prediction and automated tuning of randomized and parametric algorithms Proceedings of the 12th International Conference on Principles and Practice of Constraint Programming (CPE'06) LNCS vol. 4204 2006 Springer-Verlag 213 228 (Pubitemid 44618741)
51. F. Hutter, H. Hoos, and K. Leyton-Brown Bayesian optimization with censored response data ArXiv e-prints arXiv:1310.1947 [cs.AI] 2013
52. F. Hutter, H.H. Hoos, and K. Leyton-Brown Automated configuration of mixed integer programming solvers Proceedings of the International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (CPAIORE'10) LNCS vol. 6140 2010 Springer-Verlag 186 202
53. F. Hutter, H.H. Hoos, and K. Leyton-Brown Tradeoffs in the empirical evaluation of competing algorithm designs Special Issue on Learning and Intelligent Optimization Ann. Math. Artif. Intell. 60 1 2010 65 89
54. F. Hutter, H.H. Hoos, and K. Leyton-Brown Bayesian optimization with censored response data NIPS 2011 Workshop on Bayesian Optimization, Sequential Experimental Design, and Bandits 2011 published online
55. F. Hutter, H.H. Hoos, and K. Leyton-Brown Sequential model-based optimization for general algorithm configuration Proceedings of the 5th Workshop on Learning and Intelligent Optimization (LIONE'11) LNCS vol. 6683 2011 Springer-Verlag 507 523
56. F. Hutter, H.H. Hoos, and K. Leyton-Brown Parallel algorithm configuration Proceedings of the 6th Workshop on Learning and Intelligent Optimization (LIONE'12) LNCS 2012 Springer-Verlag 55 70 http://www.springer.com/ computer/theoretical+computer+science/book/978-3-642-44972-7
57. F. Hutter, H.H. Hoos, and K. Leyton-Brown Identifying key algorithm parameters and instance features using forward selection Proc. of LION-7 2013 in press
58. F. Hutter, H.H. Hoos, K. Leyton-Brown, and K.P. Murphy An experimental investigation of model-based parameter optimisation: SPO and beyond Proceedings of the Genetic and Evolutionary Computation Conference (GECCOE'09) 2009 271 278
59. F. Hutter, H.H. Hoos, K. Leyton-Brown, and K.P. Murphy Time-bounded sequential parameter optimization Proceedings of the 4th Workshop on Learning and Intelligent Optimization (LION'10) LNCS vol. 6073 2010 Springer-Verlag 281 298
60. F. Hutter, D.A.D. Tompkins, and H.H. Hoos Scaling and probabilistic smoothing: efficient dynamic local search for SAT Proceedings of the 8th International Conference on Principles and Practice of Constraint Programming (CP'02) LNCS vol. 2470 2002 Springer-Verlag 233 248
61. International Business Machines Corp IBM ILOG CPLEX Optimizer - Data Sheet ftp://public.dhe.ibm.com/common/ssi/ecm/en/wsd14044usen/WSD14044USEN.PDF 2012 Last accessed on August 6, 2012
62. International Business, Machines Corp, CPLEX 12.1. http://www-01.ibm.com/ software/integration/optimization/cplex-optimizer/, 2012. Last accessed on August 6, 2012.
63. D.S. Johnson, Random TSP generators for the DIMACS TSP challenge. http://www2.research.att.com/~dsj/chtsp/codes.tar, 2011. Last accessed on May 16, 2011.
64. D.R. Jones, C.D. Perttunen, and B.E. Stuckman Lipschitzian optimization without the Lipschitz constant J. Optim. Theory Appl. 79 1 1993 157 181
65. D.R. Jones, M. Schonlau, and W.J. Welch Efficient global optimization of expensive black box functions J. Glob. Optim. 13 1998 455 492 (Pubitemid 128507405)
66. T. Jones, and S. Forrest Fitness distance correlation as a measure of problem difficulty for genetic algorithms Proceedings of the 6th International Conference on Genetic Algorithms (ICGA'95) 1995 184 192
67. S. Kadioglu, Y. Malitsky, M. Sellmann, and K. Tierney ISAC - instance specific algorithm configuration Proceedings of the 19th European Conference on Artificial Intelligence (ECAI'10) 2010 751 756
68. P. Kilby, J. Slaney, S. Thiebaux, and T. Walsh Estimating search tree size Proceedings of the 21st National Conference on Artificial Intelligence (AAAI'06) 2006 1014 1019 (Pubitemid 44705409)
69. D. Knuth Estimating the efficiency of backtrack programs Math. Comput. 29 129 1975 121 136
70. L. Kotthoff, I.P. Gent, and I. Miguel An evaluation of machine learning in algorithm selection for search problems AI Commun. 25 3 2012 257 270
71. D.G. Krige A statistical approach to some basic mine valuation problems on the Witwatersrand J. Chem. Metall. Min. Soc. S. Afr. 52 6 1951 119 139
72. N.D. Lawrence, M. Seeger, and R. Herbrich Fast sparse Gaussian process methods: the informative vector machine Proceedings of the 15th Conference on Advances in Neural Information Processing Systems (NIPS'02) 2003 609 616
73. K. Leyton-Brown, H.H. Hoos, F. Hutter, and L. Xu Understanding the empirical hardness of NP-complete problems Commun. ACM 2013 in press
74. K. Leyton-Brown, E. Nudelman, G. Andrew, J. McFadden, and Y. Shoham Boosting as a metaphor for algorithm design Proceedings of the 9th International Conference on Principles and Practice of Constraint Programming (CP'03) LNCS vol. 2833 2003 Springer-Verlag 899 903
75. K. Leyton-Brown, E. Nudelman, and Y. Shoham Learning the empirical hardness of optimization problems: the case of combinatorial auctions Proceedings of the 8th International Conference on Principles and Practice of Constraint Programming (CP'02) LNCS vol. 2470 2002 Springer-Verlag 556 572
76. K. Leyton-Brown, E. Nudelman, and Y. Shoham Empirical hardness models: methodology and a case study on combinatorial auctions J. ACM 56 4 2009 1 52
77. K. Leyton-Brown, M. Pearson, and Y. Shoham Towards a universal test suite for combinatorial auction algorithms EC '00: Proceedings of the 2nd ACM Conference on Electronic Commerce 2000 66 76
78. S. Lin, and B.W. Kernighan An effective heuristic algorithm for the traveling-salesman problem Oper. Res. 21 2 1973 498 516
79. L. Lobjois, and M. Lemaître Branch and bound algorithm selection by performance prediction Proceedings of the 15th National Conference on Artificial Intelligence (AAAI'98) 1998 353 358
80. Y.S. Mahajan, Z. Fu, and S. Malik Zchaff2004: an efficient SAT solver Proceedings of the 8th International Conference on Theory and Applications of Satisfiability Testing (SAT'05) LNCS vol. 3542 2005 Springer-Verlag 360 375 (Pubitemid 41422894)
81. N. Meinshausen Quantile regression forests J. Mach. Learn. Res. 7 2006 983 999 (Pubitemid 43902307)
82. O. Mersmann, B. Bischl, H. Trautmann, M. Wagner, J. Bossek, and F. Neumann A novel feature-based approach to characterize algorithm performance for the traveling salesperson problem Ann. Math. Artif. Intell. 2013 10.1007/s10472-013-9341-2 32 p., published online: 28 March 2013
83. D. Mitchell, B. Selman, and H. Levesque Hard and easy distributions of SAT problems Proceedings of the 10th National Conference on Artificial Intelligence (AAAI'92) 1992 459 465 (Pubitemid 23633548)
84. I.T. Nabney NETLAB: Algorithms for Pattern Recognition 2002 Springer
85. V. Nannen, and A.E. Eiben Relevance estimation and value calibration of evolutionary algorithm parameters Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI'07) 2007 975 980
86. W. Nelson Applied Life Data Analysis Wiley Series in Probability and Statistics 2003 John Wiley & Sons
87. J. Nocedal, and S.J. Wright Numerical Optimization second edition 2006 Springer
88. G.R. Nudd, D.J. Kerbyson, E. Papaefstathiou, S.C. Perry, J.S. Harper, and D.V. Wilcox Pace-a toolset for the performance prediction of parallel and distributed systems Int. J. High Perform. Comput. Appl. 14 3 2000 228 251 (Pubitemid 32025043)
89. E. Nudelman, K. Leyton-Brown, G. Andrew, C. Gomes, J. McFadden, B. Selman, Y. Shoham, Satzilla 0.9. Solver description, 2003 SAT Competition, 2003.
90. E. Nudelman, K. Leyton-Brown, H.H. Hoos, A. Devkar, and Y. Shoham Understanding random SAT: beyond the clauses-to-variables ratio Proceedings of the 10th International Conference on Principles and Practice of Constraint Programming (CP'04) LNCS vol. 3258 2004 Springer-Verlag 438 452
91. B. Pfahringer, H. Bensusan, and C. Giraud-Carrier Meta-learning by landmarking various learning algorithms Proceedings of the 17th International Conference on Machine Learning (ICML'00) 2000 743 750
92. M.R. Prasad, A. Biere, and A. Gupta A survey of recent advances in SAT-based formal verification Int. J. Softw. Tools Technol. Transf. 7 2 2005 156 173 (Pubitemid 41349074)
93. J. Quinonero-Candela, C.E. Rasmussen, and C.K. Williams Approximation methods for Gaussian process regression Léon Bottou, Olivier Chapelle, Dennis DeCoste, Jason Weston, Large-Scale Kernel Machines 2007 MIT Press 203 223
94. C.E. Rasmussen, and C.K.I. Williams Gaussian Processes for Machine Learning 2006 MIT Press
95. J.R. Rice The algorithm selection problem Adv. Comput. 15 1976 65 118
96. E. Ridge, and D. Kudenko Tuning the performance of the MMAS heuristic Proceedings of the International Workshop on Engineering Stochastic Local Search Algorithms (SLS2007) LNCS vol. 4638 2007 Springer-Verlag 46 60
97. M. Roberts, and A. Howe Learned models of performance for many planners ICAPS 2007 Workshop AI Planning and Learning 2007
98. J. Sacks, W.J. Welch, T.J. Welch, and H.P. Wynn Design and analysis of computer experiments Stat. Sci. 4 4 1989 409 423
99. T.J. Santner, B.J. Williams, and W.I. Notz The Design and Analysis of Computer Experiments 2003 Springer
100. J. Schmee, and G.J. Hahn A simple method for regression analysis with censored data Technometrics 21 4 1979 417 432 (Pubitemid 10427358)
101. M. Schmidt minfunc http://www.di.ens.fr/~mschmidt/Software/minFunc.html 2012 Last accessed on August 5, 2012
102. M.R. Segal Regression trees for censored data Biometrics 44 1 1988 35 47 (Pubitemid 18098810)
103. J. Sherman, and W.J. Morrison Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix (abstract) Ann. Math. Stat. 20 1949 621
104. K. Smith-Miles Cross-disciplinary perspectives on meta-learning for algorithm selection ACM Comput. Surv. 41 1 2009 6:1 6:25
105. K. Smith-Miles, and L. Lopes Measuring instance difficulty for combinatorial optimization problems Comput. Oper. Res. 39 5 2012 875 889
106. K. Smith-Miles, and T. Tan Measuring algorithm footprints in instance space Proceedings of the 2012 Congress on Evolutionary Computation (CEC12) 2012 3446 3453
107. K. Smith-Miles, and J. van Hemert Discovering the suitability of optimisation algorithms by learning from evolved instances Ann. Math. Artif. Intell. 61 2011 87 104
108. K. Smith-Miles, J. van Hemert, and X.Y. Lim Understanding TSP difficulty by learning from evolved instances Proceedings of the 4th Workshop on Learning and Intelligent Optimization (LION10) LNCS vol. 6073 2010 Springer-Verlag 266 280
109. M. Soos CryptoMiniSat 2.5.0. Solver description SAT Race 2010
110. V. Tresp A Bayesian committee machine Neural Comput. 12 11 2000 2719 2741
111. R. Vilalta, and Y. Drissi A perspective view and survey of meta-learning Artif. Intell. Rev. 18 2 2002 77 95
112. W. Wei, and C.M. Li Switching between two adaptive noise mechanisms in local search for SAT. Solver description SAT competition 2009
113. E. Weinberger Correlated and uncorrelated fitness landscapes and how to tell the difference Biol. Cybern. 63 1990 325 336 (Pubitemid 20289470)
114. N.A. Weiss A Course in Probability 2005 Addison-Wesley
115. L. Xu, H.H. Hoos, and K. Leyton-Brown Hierarchical hardness models for SAT Proceedings of the 13th International Conference on Principles and Practice of Constraint Programming (CP07) LNCS vol. 4741 2007 Springer-Verlag 696 711
116. L. Xu, H.H. Hoos, and K. Leyton-Brown Hydra: automatically configuring algorithms for portfolio-based selection Proceedings of the 25th National Conference on Artificial Intelligence (AAAI10) 2010 210 216
117. L. Xu, F. Hutter, H. Hoos, and K. Leyton-Brown SATzilla-07: the design and analysis of an algorithm portfolio for SAT Proceedings of the 13th International Conference on Principles and Practice of Constraint Programming (CP07) LNCS vol. 4741 2007 Springer-Verlag 712 727
118. L. Xu, F. Hutter, H. Hoos, and K. Leyton-Brown SATzilla2009: an automatic algorithm portfolio for sat. Solver description SAT competition 2009
119. L. Xu, F. Hutter, H.H. Hoos, and K. Leyton-Brown SATzilla: portfolio-based algorithm selection for SAT J. Artif. Intell. Res. 32 2008 565 606
120. L. Xu, F. Hutter, H.H. Hoos, and K. Leyton-Brown Evaluating component solver contributions in portfolio-based algorithm selectors Proceedings of the 15th International Conference on Theory and Applications of Satisfiability Testing (SAT12) LNCS vol. 7317 2012 Springer-Verlag 228 241
121. L. Xu, F. Hutter, H.H. Hoos, and K. Leyton-Brown Satzilla2012: Improved algorithm selection based on cost-sensitive classification models Fifteenth International Conference on Theory and Applications of Satisfiability Testing, SAT Challenge 2012: Solver Descriptions 2012
122. E. Zarpas Benchmarking SAT solvers for bounded model checking Proceedings of the 8th International Conference on Theory and Applications of Satisfiability Testing (SAT05) LNCS vol. 3569 2005 Springer-Verlag 340 354 (Pubitemid 41425517)