Scalable Bayesian optimization using deep neural networks

32nd International Conference on Machine Learning, ICML 2015

Volumn 3, Issue , 2015, Pages 2161-2170

  Snoek, Jasper ^a   Ripped, Oren ^a,b   Swersky, Kevin ^c   Kiros, Ryan ^c   Satish, Nadathur ^d   Sundaram, Narayanan ^d   Patwary, Md Mostofa Ali ^d   Prabhat, ^e   Adams, Ryan P ^a  

^a HARVARD UNIVERSITY   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c UNIVERSITY OF TORONTO   (Canada)

^d Intel Research Cambridge Lab   (United Kingdom)

^e LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; DISTRIBUTION FUNCTIONS; FUNCTION EVALUATION; GLOBAL OPTIMIZATION; LEARNING SYSTEMS; OBJECT RECOGNITION;

ADAPTIVE BASIS FUNCTION; BAYESIAN OPTIMIZATION; COMPETITIVE MODELS; CONVOLUTIONAL NETWORKS; DEEP NEURAL NETWORKS; DEGREE OF PARALLELISM; HYPER-PARAMETER OPTIMIZATIONS; MODELING DISTRIBUTIONS;

BAYESIAN NETWORKS;

EID: 84970022032     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (58)

1. Bardenet, R., Brendel, M., Kégl, B., and Sebag, M. Collaborative hyperparameter tuning. In ICML, 2013.
2. Bergstra, J. and Bengio, Y. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281-305, 2012.
3. Bergstra, J. S., Bardenet, R., Bengio, Y., and Kegl, B. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems. 2011.
4. Bishop, C. M. Pattern Recognition and Machine Learning. Springer-Verlag New York, Inc., 2006.
5. Brochu, E., Brochu, T., and de Freitas, N. A Bayesian interactive optimization approach to procedural animation design. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2010.
6. Bull, A. D. Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research, (3-4):2879-2904, 2011.
7. Buntine, W. L. and Weigend, A. S. Bayesian back-propagation. Complex systems, 5(6):603-643, 1991.
8. Calandra, R., Peters, J., Rasmussen, C. E., and Deisenroth, M. P. Manifold Gaussian processes for regression, preprint arXiv:1402.5876, 2014a.
9. Calandra, R., Peters, J., Seyfarth, A., and Deisenroth, M. P. An experimental evaluation of Bayesian optimization on bipedal locomotion. In International Conference on Robotics and Automation, 2014b.
10. Carvalho, C. M., Poison, N. G., and Scott, J. G. Handling sparsity via the horseshoe. In Artificial Intelligence and Statistics, 2009.
11. De Freitas, J. F. Bayesian methods for neural networks. PhD thesis, Trinity College, University of Cambridge, 2003.
12. de Freitas, N., Smola, A. J., and Zoghi, M. Exponential regret bounds for Gaussian process bandits with deterministic observations. In ICML, 2012.
13. Djolonga, J., Krause, A., and Cevher, V. High dimensional Gaussian process bandits. In Advances in Neural Information Processing Systems, 2013.
14. Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H., and Leyton-Brown, K. Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice, 2013.
15. Feurer, M., Springenberg, T., and Hutter, F. Initializing Bayesian hyperparameter optimization via meta-learning. In AAAI Conference on Artificial Intelligence, 2015.
16. Garnett, R., Osborne, M. A., and Roberts, S. J. Bayesian optimization for sensor set selection. In International Conference on Information Processing in Sensor Networks, 2010.
17. Gelbart, M. A., Snoek, J., and Adams, R. P. Bayesian optimization with unknown constraints. In Uncertainty in Artificial Intelligence, 2014.
18. Ginsbourger, D. and Riche, R. L. Dealing with asyn-chronicity in parallel Gaussian process based global optimization. http://hal.archives-ouvertes.fr/ha 1-00507632, 2010.
19. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A. C., and Bengio, Y. Maxout networks. In ICML, 2013.
20. Gramacy, R. B. and Lee, H. K. H. Optimization under unknown constraints, 2010. arXiv: 1004.4027.
21. Hensman, J., Fusi, N., and Lawrence, N. Gaussian processes for big data. In Uncertainty in Artificial Intelligence, 2013.
22. Hinton, G. E. and van Camp, D. Keeping neural networks simple by minimizing the description length of the weights. In ACM Conference on Computational Learning Theory, 1993.
23. Hinton, G. E. and Salakhutdinov, R. Using deep belief nets to learn covariance kernels for Gaussian processes. In Advances in neural information processing systems, pp. 1249-1256, 2008.
24. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0580, 2012.
25. Hoffman, M., Brochu, E., and de Freitas, N. Portfolio allocation for Bayesian optimization. In Uncertainty in Artificial Intelligence, 2011.
26. Hutter, F., Hoos, H. H., and Leyton-Brown, K. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization 5, 2011.
27. Jones, D. R. A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization, 21, 2001.
28. Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
29. Kiros, R., Salakhutdinov, R., and Zemel, R. S. Multimodal neural language models. In ICML, 2014.
30. Krause, A. and Ong, C. S. Contextual Gaussian process bandit optimization. In Advances in Neural Information Processing Systems, 2011.
31. Kushner, H. J. A new method for locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, 86, 1964.
32. Lázaro-Gredilla, M. and Figueiras-Vidal, A. R. Marginalized neural network mixtures for large-scale regression. Neural Networks, IEEE Transactions on, 21(8): 1345-1351, 2010.
33. Lee, C.-Y., Xie, S., Gallagher, P., Zhang, Z., and Tu, Z. Deeply supervised nets. In Deep Learning and Representation Learning Workshop, NIPS, 2014.
34. Lin, M., Chen, Q., and Yan, S. Network in network. CoRR, abs/1312.4400, 2013. URL http ://arxiv.org/abs/1312.4400.
35. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common objects in context. In ECCV 2014, pp. 740-755. Springer, 2014.
36. Lizotte, D. Practical Bayesian Optimization. PhD thesis, University of Alberta, Edmonton, Alberta, 2008.
37. MacKay, D. J. A practical Bayesian framework for backpropagation networks. Neural computation, 4(3):448-472, 1992.
38. Mahendran, N., Wang, Z., Hamze, E, and de Freitas, N. Adaptive MCMC with Bayesian optimization. In Artificial Intelligence and Statistics, 2012.
39. Mnih, A. and Gregor, K. Neural variational inference and learning in belief networks. In ICML, 2014.
40. Mockus, J., Tiesis, V., and Zilinskas, A. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2, 1978.
41. Neal, R. Slice sampling. Annals of Statistics, 31:705-767, 2000.
42. Neal, R. M. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
43. Nickson, T., Osborne, M. A., Reece, S., and Roberts, S. Automated machine learning using stochastic algorithm tuning. NIPS Workshop on Bayesian Optimization, 2014.
44. Osborne, M. A., Garnett, R., and Roberts, S. J. Gaussian processes for global optimization. In Learning and Intelligent Optimization, 2009.
45. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic back-propagation and variational inference in deep latent Gaussian models. In ICML, 2014.
46. Snelson, E. and Ghahramani, Z. Sparse Gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems, pp. 1257-1264, 2005.
47. Snoek, J. Bayesian Optimization and Semiparametric Models with Applications to Assistive Technology. PhD thesis, University of Toronto, Toronto, Canada, 2013.
48. Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, 2012.
49. Snoek, J., Swersky, K., Zemel, R. S., and Adams, R. P. Input warping for Bayesian optimization of non-stationary functions. In ICML, 2014.
50. Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014. URL http://arxiv.org/abs/1412. 6806.
51. Srinivas, N., Krause, A., Kakade, S., and Seeger, M. Gaussian process optimization in the bandit setting: no regret and experimental design. In ICML, 2010.
52. Swersky, K, Snoek, J., and Adams, R. P. Multi-task Bayesian optimization. In Advances in Neural Information Processing Systems, 2013.
53. Titsias, M. K. Variational learning of inducing variables in sparse Gaussian processes. In International Conference on Artificial Intelligence and Statistics, pp. 567-574, 2009.
54. Wan, L., Zeiler, M. D., Zhang, S., Le Cun, Y., and Fergus, R. Regularization of neural networks using dropconnect. In ICML, 2013.
55. Wang, Z., Zoghi, M., Hutter, F., Matheson, D., and de Freitas, N. Bayesian optimization in high dimensions via random embeddings. In IJCAI, 2013.
56. Williams, C. K. I. Computing with infinite networks. In Advances in Neural Information Processing Systems, 1996.
57. Xu, K., Ba, J., Kiros, R., Cho, K, Courville, A., Salakhutdinov, R., Zemel, R., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044v2, 2015.
58. Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv preprint arXiv:1207.0580, 2015.