Random search for hyper-parameter optimization

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 281-305

  Bergstra, James ^a   Bengio, Yoshua ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Deep learning; Global optimization; Model selection; Neural networks; Response surface modeling

Indexed keywords

COMPUTATION TIME; COMPUTATIONAL BUDGET; CONFIGURATION SPACE; DATA SETS; DEEP BELIEF NETWORKS; DEEP LEARNING; EMPIRICAL EVIDENCE; GAUSSIAN PROCESSES; GRID SEARCH; HIERARCHICAL MODEL; HIGH THROUGHPUT; HYPER-PARAMETER; MODEL SELECTION; OPTIMIZATION ALGORITHMS; RANDOM SEARCHES; RESPONSE SURFACE MODELING;

ALGORITHMS; BAYESIAN NETWORKS; GLOBAL OPTIMIZATION; HIERARCHICAL SYSTEMS; NEURAL NETWORKS; WAVE FUNCTIONS;

OPTIMIZATION;

EID: 84857855190     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (34)

1. I. A. Antonov and V. M. Saleev. An economic method of computing LPt-sequences. USSR Computational Mathematics and Mathematical Physics, 19(1):252-256, 1979.
2. R. Bellman. Adaptive Control Processes: A Guided Tour. Princeton University Press, New Jersey, 1961.
3. Y. Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1): 1-127, 2009. doi: 10.1561/2200000006.
4. Y. Bengio and X. Glorot. Understanding the difficulty of training deep feedforward neural networks. In Y. W. Teh and M. Titterington, editors, Proc. of The Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS'10), pages 249-256, 2010.
5. J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, and Y. Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral.
6. C. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, London, UK, 1995.
7. P. Bratley, B. L. Fox, and H. Niederreiter. Implementation and tests of low-discrepancy sequences. Transactions on Modeling and Computer Simulation, (TOMACS), 2(3):195-213, 1992.
8. R. E. Caflisch,W.Morokoff, and A. Owen. Valuation of mortgage backed securities using brownian bridges to reduce effective dimension, 1997.
9. C. Chang and C. Lin. LIBSVM: A Library for Support Vector Machines, 2001.
10. I. Czogiel, K. Luebke, and C. Weihs. Response surface methodology for optimizing hyper parameters. Technical report, Universität Dortmund Fachbereich Statistik, September 2005.
11. S. S. Drew and T. Homem de Mello. Quasi-Monte Carlo strategies for stochastic optimization. In Proc. of the 38th Conference on Winter Simulation, pages 774 - 782, 2006.
12. D. Erhan, Y. Bengio, A. Courville, P. Manzagol, P. Vincent, and S. Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11:625-660, 2010.
13. J. H. Halton. On the efficiency of certain quasi-random sequences of points in evaluating multidimensional integrals. Numerische Mathematik, 2:84-90, 1960.
14. N. Hansen, S. D. Müller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11 (1):1-18, 2003. (Pubitemid 37044186)
15. G. E. Hinton. A practical guide to training restricted Boltzmann machines. Technical Report 2010-003, University of Toronto, 2010. version 1.
16. G. E. Hinton, S. Osindero, and Y. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006. (Pubitemid 44024729)
17. F. Hutter. Automated Configuration of Algorithms for Solving Hard Computational Problems. PhD thesis, University of British Columbia, 2009.
18. F. Hutter, H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In LION-5, 2011. Extended version as UBC Tech report TR-2010-10.
19. S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science, 220 (4598):671-680, 1983.
20. N. L. Kleinman, J. C. Spall, and D. Q. Naiman. Simulation-based optimization with stochastic approximation using common random numbers. Management Science, 45(11):1570-1578, November 1999. doi: doi:10.1287/mnsc.45. 11.1570.
21. H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Z. Ghahramani, editor, Proceedings of the Twenty-fourth International Conference on Machine Learning (ICML'07), pages 473-480. ACM, 2007.
22. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, November 1998a.
23. Y. LeCun, L. Bottou, G. Orr, and K. Muller. Efficient backprop. In G. Orr and K. Muller, editors, Neural Networks: Tricks of the Trade. Springer, 1998b.
24. M. Galassi et al. GNU Scientific Library Reference Manual, 3rd edition, 2009.
25. M. D. McKay, R. J. Beckman, and W. J. Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2): 239-245, May 1979. doi: doi:10.2307/1268522. (Pubitemid 9452235)
26. A. Nareyek. Choosing search heuristics by non-stationary reinforcement learning. Applied Optimization, 86:523-544, 2003.
27. R. M. Neal. Assessing relevance determination methods using DELVE. In C. M. Bishop, editor, Neural Networks and Machine Learning, pages 97-129. Springer-Verlag, 1998.
28. J. A. Nelder and R. Mead. A simplex method for function minimization. The Computer Journal, 7: 308-313, 1965.
29. M. J. D. Powell. A direct search optimization method that models the objective and constraint functions by linear interpolation. Advances in Optimization and Numerical Analysis, pages 51-67, 1994.
30. C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
31. Ingo Rechenberg. Evolutionsstrategie - Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Fommann-Holzboog, Stuttgart, 1973.
32. A. Srinivasan and G. Ramakrishnan. Parameter screening and optimisation for ILP using designed experiments. Journal of Machine Learning Research, 12:627-662, February 2011.
33. P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol. Extracting and composing robust features with denoising autoencoders. In W. W. Cohen, A. McCallum, and S. T. Roweis, editors, Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML'08), pages 1096-1103. ACM, 2008.
34. T. Weise. Global Optimization Algorithms - Theory and Application. Self-Published, second edition, 2009. Online available at http://www.it-weise. de/.