Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn Part F128815, Issue , 2013, Pages 847-855

  Thornton, Chris ^a   Hutter, Frank ^a   Hoos, Holger H ^a   Leyton Brown, Kevin ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Hyperparameter optimization; Model selection; Weka

Indexed keywords

ARTIFICIAL INTELLIGENCE; CLASSIFICATION (OF INFORMATION); DATA MINING; EDUCATION; LEARNING SYSTEMS; OPTIMIZATION;

BAYESIAN OPTIMIZATION; CLASSIFICATION ALGORITHM; CLASSIFICATION APPROACH; CLASSIFICATION PERFORMANCE; HYPER-PARAMETER OPTIMIZATIONS; MODEL SELECTION; STANDARD DISTRIBUTIONS; WEKA;

LEARNING ALGORITHMS;

EID: 85018371540     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2487575.2487629     Document Type: Conference Paper

References (34)

1. M. Adankon and M. Cheriet. Model selection for the LS-SVM. application to handwriting recognition. Pattern Recognition, 42(12):3264-3270, 2009.
2. R. Bardenet, M. Brendel, B. Kégl, and M. Sebag. Collaborative hyperparameter tuning. In Proc. of ICML-13, 2013.
3. Y. Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12(8):1889-1900, 2000.
4. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for Hyper-Parameter Optimization. In Proc. of NIPS-11, 2011.
5. J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. JMLR, 13:281-305, 2012.
6. A. Biem. A model selection criterion for classification: Application to HMM topology optimization. In Proc. of ICDAR-03, pages 104-108. IEEE, 2003.
7. H. Bozdogan. Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3):345-370, 1987.
8. P. Brazdil, C. Soares, and J. Da Costa. Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3):251-277, 2003.
9. E. Brochu, V. M. Cora, and N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Technical Report UBC TR-2009-23 and arXiv:1012.2599v1, Department of Computer Science, University of British Columbia, 2009.
10. O. Chapelle, V. Vapnik, and Y. Bengio. Model selection for small sample regression. Machine Learning, 2001.
11. T. Desautels, A. Krause, and J. Burdick. Parallelizing exploration-exploitation tradeoffs with gaussian process bandit optimization. In Proc. of ICML-12, 2012.
12. A. Frank and A. Asuncion. UCI machine learning repository, 2010. URL: http://archive.ics.uci.edu/ml University of California, Irvine, School of Information and Computer Sciences.
13. X. Guo, J. Yang, C. Wu, C. Wang, and Y. Liang. A novel LS-SVMs hyper-parameter selection based on particle swarm optimization. Neurocomputing, 71(16):3211-3215, 2008.
14. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. Witten. The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11(1):10-18, 2009.
15. F. Hutter, H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Proc. of LION-5, pages 507-523, 2011.
16. F. Hutter, H. Hoos, K. Leyton-Brown, and T. Stützle. ParamILS: An automatic algorithm configuration framework. JAIR, 36(1):267-306, 2009.
17. F. Hutter, H. H. Hoos, and K. Leyton-Brown. Parallel algorithm configuration. In Proc. of LION-6, pages 55-70, 2012.
18. D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black box functions. Journal of Global Optimization, 13:455-492, 1998.
19. R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proc. of IJCAI-95, pages 1137-1145, 1995.
20. A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
21. R. Leite, P. Brazdil, and J. Vanschoren. Selecting classification algorithms with active testing. In Proc. of MLDM-12, pages 117-131, 2012.
22. M. López-Ibáñez, J. Dubois-Lacoste, T. Stützle, and M. Birattari. The irace package, iterated race for automatic algorithm configuration. Technical Report TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium, 2011.
23. O. Maron and A. Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. In Proc. of NIPS-94, pages 59-66, 1994.
24. A. McQuarrie and C. Tsai. Regression and time series model selection. World Scientific, 1998.
25. B. Pfahringer, H. Bensusan, and C. Giraud-Carrier. Meta-learning by landmarking various learning algorithms. In Proc. of ICML-00, pages 743-750, 2000.
26. T. Schaul, J. Bayer, D. Wierstra, Y. Sun, M. Felder, F. Sehnke, T. Rückstieß, and J. Schmidhuber. PyBrain. JMLR, 2010.
27. M. Schonlau, W. J. Welch, and D. R. Jones. Global versus local search in constrained optimization of computer models. In N. Flournoy, W. Rosenberger, and W. Wong, editors, New Developments and Applications in Experimental Design, volume 34, pages 11-25. Institute of Mathematical Statistics, Hayward, California, 1998.
28. J. Snoek, H. Larochelle, and R. Adams. Opportunity cost in Bayesian optimization. In NIPS Workshop on Bayesian Optimization, Sequential Experimental Design, and Bandits, 2011. Published online.
29. J. Snoek, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Proc. of NIPS-12, 2012.
30. N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proc. of ICML-10, pages 1015-1022, 2010.
31. V. Strijov and G. Weber. Nonlinear regression model generation using hyperparameter optimization. Computers & Mathematics with Applications, 60(4):981-988, 2010.
32. R. Vilalta and Y. Drissi. A perspective view and survey of meta-learning. Artif. Intell. Rev., 18(2):77-95, Oct. 2002.
33. L. Xu, H. H. Hoos, and K. Leyton-Brown. Hydra: Automatically configuring algorithms for portfolio-based selection. In Proc. of AAAI-10, pages 210-216, 2010.
34. P. Zhao and B. Yu. On model selection consistency of lasso. JMLR, 7:2541-2563, Dec. 2006.