Google vizier: A service for black-box optimization

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn Part F129685, Issue , 2017, Pages 1487-1496

  Golovin, Daniel ^a   Solnik, Benjamin ^a   Moitra, Subhodeep ^a   Kochanski, Greg ^a   Karro, John ^a   Sculley, D ^a  

^a GOOGLE INC   (United States)

Author keywords

Automated stopping; Bayesian optimization; Black box optimization; Gaussian processes; Hyperparameters; Transfer learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; DATA MINING; LEARNING SYSTEMS;

AUTOMATED STOPPING; BAYESIAN OPTIMIZATION; BLACK-BOX OPTIMIZATION; GAUSSIAN PROCESSES; HYPERPARAMETERS; TRANSFER LEARNING;

OPTIMIZATION;

EID: 85029112253     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/3097983.3098043     Document Type: Conference Paper

References (33)

1. Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michele Sebag. 2013. Collaborative hyperparameter tuning. ICML 2 (2013), 199.
2. James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems. 2546-2554.
3. J Bernardo, MJ Bayarri, JO Berger, AP Dawid, D Heckerman, AFM Smith, and M West. 2011. Optimization under unknown constraints. Bayesian Statistics 9 9 (2011), 229.
4. 3 data-driven documents. IEEE transactions on visualization and computer graphics 17, 12 (2011), 2301-2309.
5. Herman Chernoff. 1959. Sequential Design of Experiments. Ann. Math. Statist. 30, 3 (09 1959), 755-770. https://doi.org/10.1214/aoms/1177706205
6. Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. 2017. Capacity and Trainability in Recurrent Neural Networks. In Profeedings of the International Conference on Learning Representations (ICLR).
7. Andrew R Conn, Katya Scheinberg, and Luis N Vicente. 2009. Introduction to derivative-free optimization. SIAM.
8. Thomas Desautels, Andreas Krause, and Joel W Burdick. 2014. Parallelizing exploration-exploitation tradeoffs in Gaussian process bandit optimization. Journal of Machine Learning Research 15, 1 (2014), 3873-3923.
9. Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. In IJCAI. 3460-3468.
10. Steffen Finck, Nikolaus Hansen, Raymond Rost, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Presentation of the Noiseless Functions. http://coco.gforge.inria.fr/lib/exe/fetch.php?media=download3.6:bbobdocfunctions.pdf. (2009). [Online].
11. Jacob R Gardner, Matt J Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John P Cunningham. 2014. Bayesian Optimization with Inequality Constraints. In ICML. 937-945.
12. Michael A Gelbart, Jasper Snoek, and Ryan P Adams. 2014. Bayesian optimization with unknown constraints. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence. AUAI Press, 250-259.
13. Josep Ginebra and Murray K. Clayton. 1995. Response Surface Bandits. Journal of the Royal Statistical Society. Series B (Methodological) 57, 4 (1995), 771-784. http://www.jstor.org/stable/2345943
14. Google. 2017. Polymer: Build modern apps using web components. https://github.com/Polymer/polymer. (2017). [Online].
15. Google. 2017. Protocol Buffers: Google's data interchange format. https://github.com/google/protobuf. (2017). [Online].
16. Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation 9, 2 (2001), 159-195.
17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770-778.
18. Julian Heinrich and Daniel Weiskopf. 2013. State of the Art of Parallel Coordinates. In Eurographics (STARs). 95-116.
19. Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2011. Sequential modelbased optimization for general algorithm configuration. In International Conference on Learning and Intelligent Optimization. Springer, 507-523.
20. Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2016. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. CoRR abs/1603.06560 (2016). http://arxiv.org/abs/1603.06560
21. J Moćkus, V Tiesis, and A Źilinskas. 1978. The Application of Bayesian Methods for Seeking the Extremum. Vol. 2. Elsevier. 117-128 pages.
22. John A Nelder and Roger Mead. 1965. A simplex method for function minimization. The computer journal 7, 4 (1965), 308-313.
23. Carl Edward Rasmussen and Christopher K. I. Williams. 2005. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press.
24. Luis Miguel Rios and Nikolaos V Sahinidis. 2013. Derivative-free optimization: a review of algorithms and comparison of software implementations. Journal of Global Optimization 56, 3 (2013), 1247-1293.
25. Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando de Freitas. 2016. Taking the human out of the loop: A review of bayesian optimization. Proc. IEEE 104, 1 (2016), 148-175.
26. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems. 2951-2959.
27. Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. 2015. Scalable Bayesian Optimization Using Deep Neural Networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 (JMLR Workshop and Conference Proceedings), Francis R. Bach and David M. Blei (Eds.), Vol. 37. JMLR.org, 2171-2180. http://jmlr.org/proceedings/papers/v37/snoek15.html
28. Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. 2016. Bayesian Optimization with Robust Bayesian Neural Networks. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 4134-4142. http://papers.nips.cc/paper/6117-bayesian-optimization-with-robust-bayesian-neural-networks.pdf
29. Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. 2010. Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design. ICML (2010).
30. Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. 2014. Freeze-thaw Bayesian optimization. arXiv preprint arXiv:1406.3896 (2014).
31. Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. 2016. Deep kernel learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics. 370-378.
32. Dani Yogatama and Gideon Mann. 2014. Eficient Transfer Learning Method for Automatic Hyperparameter Tuning. JMLR: W & CP 33 (2014), 1077-1085.
33. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014).