Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves

IJCAI International Joint Conference on Artificial Intelligence

Volumn 2015-January, Issue , 2015, Pages 3460-3468

  Domhan, Tobias ^a   Springenberg, Jost Tobias ^a   Hutter, Frank ^a  

^a UNIVERSITY OF FREIBURG   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; BENCHMARKING; EXTRAPOLATION; LEARNING SYSTEMS; NETWORK ARCHITECTURE; NEURAL NETWORKS; OBJECT RECOGNITION; OPTIMIZATION; STOCHASTIC SYSTEMS;

COMPUTATIONAL RESOURCES; DEEP NEURAL NETWORKS; EARLY TERMINATION; HYPER-PARAMETER OPTIMIZATIONS; MACHINE LEARNING PROBLEM; PROBABILISTIC MODELING; STATE OF THE ART; STOCHASTIC GRADIENT DESCENT;

CURVE FITTING;

EID: 84949921865     PISSN: 10450823     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. Y. Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12(8):1889-1900, 2000.
2. J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. JMLR, 13(1):281-305, 2012.
3. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In Proc. of NIPS, pages 2546-2554, 2011.
4. J. Bergstra, D. Yamins, and D.D. Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proc. of ICML, pages 115-123, 2013.
5. L. Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
6. E. Brochu, V. M. Cora, and N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. CoRR, abs/1012.2599, 2010.
7. A. Coates, A. Y. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proc. of AISTATS, pages 215-223, 2011.
8. G. Dahl, T. Sainath, and G. Hinton. Improving deep neural networks for lvcsr using rectified linear units and dropout. In Proc. of ICASSP, pages 8609-8613. IEEE, 2013.
9. L. Deng, G. Hinton, and B. Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In Proc. of ICASSP, 2013.
10. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proc. of ICML, 2014.
11. K. Eggensperger, M. Feurer, F. Hutter, J. Bergstra, J. Snoek, H. Hoos, and K. Leyton-Brown. Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice (BayesOpt'13), 2013.
12. D. Foreman-Mackey, D. W. Hogg, D. Lang, and J. Goodman. emcee: The MCMC Hammer. PASP, 125:306-312, 2013.
13. L. Frey and D. Fisher. Modeling decision tree performance with the power law. In Proc. of AISTATS, 1999.
14. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proc. of AISTATS, pages 249-256, 2010.
15. I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In Proc. of ICML, 2013.
16. B. Gu, F. Hu, and H. Liu. Modelling classification performance for large data sets. In Proc. of WAIM, pages 317-328. Springer, 2001.
17. F. Hutter, H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Proc. of LION, pages 507-523. Springer, 2011.
18. F. Hutter, L. Xu, H. H. Hoos, and K. Leyton-Brown. Algorithm runtime prediction: Methods and evaluation. AIJ, 206(0):79-111, 2014.
19. K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In Proc. of ICCV, 2009.
20. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
21. D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
22. P. Kolachina, N. Cancedda, M. Dymetman, and S. Venkatapathy. Prediction of learning curves in machine translation. In Proc. of ACL, pages 22-30, 2012.
23. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of NIPS, pages 1097-1105, 2012.
24. A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
25. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
26. C. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply supervised nets. In Deep Learning and Representation Learning Workshop, NIPS, 2014.
27. M. Lin, Q. Chen, and S. Yan. Network in network. In ICLR: Conference Track, 2014.
28. O. Maron and A. Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. In Proc. of NIPS, pages 59-66, 1994.
29. G. Montavon, G. Orr, and K.-R. Müller, editors. Neural Networks: Tricks of the Trade - Second Edition, volume 7700 of LNCS. Springer, 2012.
30. C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
31. J. Snoek, H. Larochelle, and R.P. Adams. Practical Bayesian optimization of machine learning algorithms. In Proc. of NIPS, pages 2951-2959, 2012.
32. J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In arxiv:cs/arXiv:1412.6806, 2015.
33. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15:1929-1958, 2014.
34. K. Swersky, D. Duvenaud, J. Snoek, F. Hutter, and M. Osborne. Raiders of the lost architecture: Kernels for Bayesian optimization in conditional parameter spaces. In NIPS workshop on Bayesian Optimization in theory and practice (BayesOptâAZ13), 2013.
35. K. Swersky, J. Snoek, and R. P. Adams. Freeze-thaw Bayesian optimization. arXiv preprint arXiv:1406.3896, 2014.
36. Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-315, 2007.