Using meta-learning to initialize Bayesian optimization of hyperparameters

CEUR Workshop Proceedings

Volumn 1201, Issue , 2014, Pages 3-10

  Feurer, Matthias ^a   Springenberg, Jost Tobias ^a   Hutter, Frank ^a  

^a UNIVERSITY OF FREIBURG   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; LEARNING SYSTEMS;

BAYESIAN OPTIMIZATION; CLASSIFICATION DATASETS; COMPLEX PROBLEMS; COMPUTATIONAL OVERHEADS; HYPER-PARAMETER OPTIMIZATIONS; INITIALIZATION PROCEDURES; LEARNING FRAMEWORKS; SEQUENTIAL MODEL;

CLASSIFICATION (OF INFORMATION);

EID: 84908297240     PISSN: 16130073     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. R. Bardenet, M. Brendel, B. Kégl, and M. Sebag, 'Collaborative hyper-parameter tuning', in Proc. of ICML, (2013).
2. H. Bensusan and C. Giraud-Carrier, 'Discovering task neighbourhoods through landmark learning performances', in Proc. of 4th PKDD. Springer, (September 2000).
3. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, 'Algorithms for hyper-parameter optimization', in Proc. of NIPS, (2011).
4. J. Bergstra and Y. Bengio, 'Random search for hyper-parameter optimization', JMLR, 13, (February 2012).
5. J. Bergstra, D. Yamins, and D. D. Cox, 'Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures', in Proc. of ICML, (2013).
6. L. Breiman, 'Random forests', Machine Learning, 45, (2001).
7. E. Brochu, V. M. Cora, and N. de Freitas, 'A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning', CoRR, abs/1012.2599, (2010).
8. Chih-Chung Chang and Chih-Jen Lin, 'LIBSVM: A library for support vector machines', ACM Transactions on Intelligent Systems and Technology, 2, (2011).
9. K. Eggensperger, M. Feurer, F. Flutter, J. Bergstra, J. Snoek, H. H. Hoos, and K. Leyton-Brown, 'Towards an empirical foundation for assessing bayesian optimization of hyperparameters', in NIPS workshop on Bayesian Optimization, (2013).
10. T.A.F. Gomes, R.B.C. Prudêncio, C. Soares, A. Rossi, and A. Carvalho, 'Combining meta-learning and search techniques to select parameters for support vector machines', Neurocomputing, 75(1), (2012).
11. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and LH. Witten, 'The WEKA data mining software: an update', ACM SIGKDD Explorations Newsletter, 11(1), 10-18, (2009).
12. P. Hennig and C. Schuler, 'Entropy search for information-efficient global optimization', JMLR, 13, (2012).
13. M. W. Hoffman, B. Shahriari, and N. de Freitas, 'Exploiting correlation and budget constraints in Bayesian multi-armed bandit optimization', ArXiv e-prints, (March 2013).
14. F. Hutter, H. H. Hoos, and K. Leyton-Brown, 'Sequential model-based optimization for general algorithm configuration', in Proc. of LION-5, (2011).
15. F. Hutter, L. Xu, H. H. Hoos, and K. Leyton-Brown, 'Algorithm runtime prediction: Methods and evaluation', JAIR, 206(0), 79-111, (2014).
16. D.R. Jones, M. Schonlau, and W. Welch, 'Efficient global optimization of expensive black box functions', Journal of Global Optimization, 13, (1998).
17. A. Kalousis, Algorithm Selection via Meta-Learning. University of Geneve, Department of Computer Science, Ph.D. dissertation, University of Geneve, 2002.
18. B. Komer, J. Bergstra, and C. Eliasmith, 'Hyperopt-sklearn: Automatic hyperparameter configuration for scikit-learn', in ICML workshop on AutoML, (2014).
19. Rui Leite, Pavel Brazdil, and Joaquin Vanschoren, 'Selecting classification algorithms with active testing on similar datasets', in 5th PLAN-LEARN WORKSHOP at ECAI, (2012).
20. Machine Learning, Neural and Statistical Classification, eds., Donald Mchie, D. J. Spiegelhalter, C. C. Taylor, and John Campbell, Ellis Horwood, 1994.
21. P.B.C. Miranda, R.B.C. Prudêncio, A. Carvalho, and C. Soares, 'Combining meta-learning with multi-objective particle swarm algorithms for SVM parameter selection: An experimental analysis', in Brazilian Symposium on Neural Networks, (2012).
22. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, 'Scikit-learn: Machine learning in Python', JMLR, 12, (2011).
23. B. Pfahringer, H. Bensusan, and C. Giraud-Carrier, 'Meta-learning by landmarking various learning algorithms', in Proc. of ICML, (2000).
24. C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning, The MIT Press, 2006.
25. M. Reif, F. Shafait, and A. Dengel, 'Prediction of classifier training time including parameter optimization', in KI2011: Advances in Artificial Intelligence, (2011).
26. M. Reif, F. Shafait, and A. Dengel, 'Meta-learning for evolutionary parameter optimization of classifiers', Machine Learning, 87, (2012).
27. M. Reif, F. Shafait, and A. Dengel. Meta2-features: Providing meta-learners more information, 2012. Poster and Demo Track of the 35th German Conference on AI.
28. Bernhard Scholkopf and Alexander J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge, MA, USA, 2001.
29. J. Snoek, H. Larochelle, and R.P Adams, 'Practical bayesian optimization of machine learning algorithms', in Proc. of NIPS, (2012).
30. C. Soares and PB. Brazdil, 'Zoomed ranking: Selection of classification algorithms based on relevant performance information', in Proc. of PKDD'00, Springer, (2000).
31. N. Srinivas, A. Krause, S. Kakade, and M. Seeger, 'Gaussian process optimization in the bandit setting: No regret and experimental design', in Proc. of ICML, (2010).
32. K. Swersky, J. Snoek, and R.P. Adams, 'Multi-task bayesian optimization', in Proc. of NIPS, (2013).
33. C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms', in Proc. of KDD'13, (2013).
34. J. N. van Rijn, B. Bischl, L. Torgo, B. Gao, V. Umaashankar, S. Fischer, P. Winter, B. Wiswedel, M. R. Berthold, and J. Vanschoren, 'OpenML: a collaborative science platform', in Proc. of ECML/PKDD'13, (2013).
35. D. Yogatama and G. Mann, 'Efficient transfer learning method for automatic hyperparameter tuning', in Proc. of AISTATS, (2014).