Taking the human out of the loop: A review of Bayesian optimization

Proceedings of the IEEE

Volumn 104, Issue 1, 2016, Pages 148-175

  Shahriari, Bobak ^a   Swersky, Kevin ^b,c   Wang, Ziyu ^d,e   Adams, Ryan P ^c,f   De Freitas, Nando ^d,e,g  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

^c TWITTER INC   (United States)

^d UNIVERSITY OF OXFORD   (United Kingdom)

^e DEEPMIND   (United Kingdom)

^f HARVARD UNIVERSITY   (United States)

^g CANADIAN INSTITUTE FOR ADVANCED RESEARCH   (Canada)

Author keywords

decision making; design of experiments; genomic medicine; optimization; response surface methodology; statistical learning

Indexed keywords

APPLICATION PROGRAMS; DECISION MAKING; DESIGN OF EXPERIMENTS; DIGITAL STORAGE; OPTIMIZATION;

BAYESIAN OPTIMIZATION; BIG DATA APPLICATIONS; COMPLEX SOFTWARE SYSTEMS; CONFIGURATION PARAMETERS; HETEROGENEOUS COMPUTING; METHODOLOGICAL ASPECTS; RESPONSE SURFACE METHODOLOGY; STATISTICAL LEARNING;

REAL TIME SYSTEMS;

EID: 84949985138     PISSN: 00189219     EISSN: 15582256     Source Type: Journal    
DOI: 10.1109/JPROC.2015.2494218     Document Type: Review

References (163)

1. R. P. Adams and O. Stegle. Gaussian process product models for nonparametric nonstationarity. in Proc. Int. Conf. Mach. Learn., 2008, pp. 1-8
2. S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. in Proc. Int. Conf. Mach. Learn., 2013, pp. 127-135
3. E. B. Anderes and M. L. Stein. Estimating deformations of isotropic Gaussian random fields on the plane. Ann. Stat., vol. 36, no. 2, pp. 719-741, 2008
4. C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan. An introduction to MCMC for machine learning. Mach. Learn., vol. 50, no. 1-2, pp. 5-43, 2003
5. J.-A. M. Assael, Z. Wang, and N. De Freitas. Heteroscedastic treed Bayesian optimisation. 2014. [Online]. Available: http://abs/abs/1410.7172
6. C. Audet, J. J. Dennis, D. W. Moore, A. Booker, and P. D. Frank. Surrogatemodel-based method for constrained optimization. in Proc. AIAA/USAF/NASA/ISSMO Symp. Multidisciplinary Anal. Optim., 2000, DOI: 10.2514/6.2000-4891
7. J. Audibert, S. Bubeck, and R. Munos. Best arm identification in multi-Armed bandits. in Proc. Conf. Learn. Theory, 2010, pp. 41-53
8. P. Auer. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res., vol. 3, pp. 397-422, 2003
9. P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. Gambling in a rigged casino: The adversarial multi-Armed bandit problem. in Proc. Symp. Found. Comput. Sci., 1995, pp. 322-331
10. P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput., vol. 32, no. 1, pp. 48-77, 2002
11. J. Azimi, A. Jalali, and X. Fern. Hybrid batch Bayesian optimization. in Proc. Int. Conf. Mach. Learn., 2012, pp. 1215-1222
12. R. Bardenet, M. Brendel, B. Kégl, and M. Sebag. Collaborative hyperparameter tuning. in Proc. Int. Conf. Mach. Learn., 2013, pp. 199-207
13. R. Bardenet and B. Kégl. Surrogating the surrogate: Accelerating Gaussian-processbased global optimization with a mixture cross-entropy algorithm. in Proc. Int. Conf. Mach. Learn., 2010, pp. 55-62
14. T. Bartz-Beielstein, C. Lasarczyk, and M. Preuss. Sequential parameter optimization. in Proc. IEEE Congr. Evol. Comput., 2005, pp. 773-780
15. R. Benassi, J. Bect, and E. Vazquez. Robust Gaussian process-based global optimization using a fully Bayesian expected improvement criterion. in Learning and Intelligent Optimization, vol. 6683, C. Coello, Ed. Berlin, Germany: Springer-Verlag, 2011, pp. 176-190
16. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 2546-2554
17. J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., vol. 13, pp. 281-305, 2012
18. J. Bergstra, B. Komer, C. Eliasmith, and D. Warde-Farley. Preliminary evaluation of hyperopt algorithms on HPOLib. in Proc. Int. Conf. Mach. Learn. AutoML Workshop, 2014
19. J. Bergstra, D. Yamins, and D. D. Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. in Proc. Int. Conf. Mach. Learn., 2013, pp. 115-123
20. S. Bochner, Lectures on Fourier Integrals. Princeton, NJ, USA: Princeton Univ. Press, 1959
21. E. V. Bonilla, K. M. A. Chai, and C. K. I. Williams. Multi-Task Gaussian process prediction. in Proc. Adv. Neural Inf. Process. Syst., 2008, pp. 153-160
22. L. Bornn, G. Shaddick, and J. V. Zidek. Modeling nonstationary processes through dimension expansion. J. Amer. Stat. Soc., vol. 107, no. 497, 2012, pp. 281-289
23. P. Boyle. Gaussian processes for regression and optimisation. Ph.D. dissertation, Victoria Univ. Wellington, Wellington, New Zealand, 2007
24. L. Breiman. Random forests. Mach. Learn., vol. 45, no. 1, pp. 5-32, 2001
25. L. Breiman, J. Friedman, R. Olshen, and C. Stone, Classification and Regression Trees. New York, NY, USA: Wadsworth and Brooks, 1984
26. E. Brochu, T. Brochu, and N. De Freitas. A Bayesian interactive optimization approach to procedural animation design. in Proc. ACM SIGGRAPH/Eurograph. Symp. Comput. Animat., 2010, pp. 103-112
27. E. Brochu, V. M. Cora, and N. De Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Dept. Comput. Sci., Univ. British Columbia, Vancouver, BC, Canada, Tech. Rep. UBC TR-2009-23, 2009
28. E. Brochu, N. De Freitas, and A. Ghosh. Active preference learning with discrete choice data. in Proc. Adv. Neural Inf. Process. Syst., 2007, pp. 409-416
29. S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-Armed bandit problems. Found. Trends Mach. Learn., vol. 5, no. 1, pp. 1-122, 2012
30. S. Bubeck, R. Munos, and G. Stoltz. Pure exploration in multi-Armed bandits problems. in Proc. Int. Conf. Algorithmic Learn. Theory, 2009, pp. 23-37
31. S. Bubeck, R. Munos, G. Stoltz, and C. Szepesvari. X-Armed bandits. J. Mach. Learn. Res., vol. 12, pp. 1655-1695, 2011
32. A. D. Bull. Convergence rates of efficient global optimization algorithms. J. Mach. Learn. Res., vol. 12, pp. 2879-2904, 2011
33. D. Busby. Hierarchical adaptive experimental design for Gaussian process emulators. Reliab. Eng. Syst. Safety, vol. 94, no. 7, pp. 1183-1193, Jul. 2009
34. R. Calandra, J. Peters, C. E. Rasmussen, and M. P. Deisenroth, Manifold Gaussian processes for regression, 2014. [Online]. Available: arXiv: 1402.5876
35. A. Carpentier and R. Munos. Bandit theory meets compressed sensing for high dimensional stochastic linear bandit. in Proc. 15th Int. Conf. Artif. Intell. Stat., 2012, pp. 190-198
36. N. Cesa-Bianchi and G. Lugosi, Prediction, Learning, Games. New York, NY, USA: Cambridge Univ. Press, 2006
37. K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Stat. Sci., vol. 10, no. 3, pp. 273-304, 1995
38. O. Chapelle and L. Li. An empirical evaluation of Thompson sampling. in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 2249-2257
39. B. Chen, R. Castro, and A. Krause. Joint optimization and variable selection of highdimensional Gaussian processes. in Proc. Int. Conf. Mach. Learn., 2012, pp. 1423-1430
40. S. Clark. Parallel machine learning algorithms in bioinformatics and global optimization. Ph.D. dissertation, Cornell Univ., Ithaca, NY, USA, 2012
41. E. Contal, V. Perchet, and N. Vayatis. Gaussian process optimization with mutual information. in Proc. Int. Conf. Mach. Learn., 2013, pp. 253-261
42. A. Criminisi, J. Shotton, and E. Konukoglu. Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning. Found. Trends Comput. Graph. Vis., vol. 7, pp. 81-227, 2011
43. N. De Freitas, A. Smola, and M. Zoghi. Exponential regret bounds for Gaussian process bandits with deterministic observations. in Proc. Int. Conf. Mach. Learn., 2012, pp. 1743-1750
44. M. Denil, L. Bazzani, H. Larochelle, and N. De Freitas. Learning where to attend with deep architectures for image tracking. Neural Comput., vol. 24, no. 8, pp. 2151-2184, 2012
45. T. Desautels, A. Krause, and J. Burdick. Parallelizing exploration-exploitation tradeoffs with Gaussian process bandit optimization. J. Mach. Learn. Res., vol. 15, pp. 4053-4103, 2014
46. J. Djolonga, A. Krause, and V. Cevher. High dimensional Gaussian process bandits. in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 1025-1033
47. T. Domhan, J. T. Springenberg, and F. Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. in Proc. 24th Int. Joint Conf. Artif. Intell., Jul. 2015, pp. 3460-3468
48. T. Domhan, T. Springenberg, and F. Hutter. Extrapolating learning curves of deep neural networks. in Proc. Int. Conf. Mach. Learn. AutoML Workshop, 2014
49. M. Feurer, T. Springenberg, and F. Hutter. Initializing Bayesian hyperparameter optimization via meta-learning. in Proc. Nat. Conf. Artif. Intell., 2015, pp. 1128-1135
50. V. Gabillon, M. Ghavamzadeh, and A. Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 3212-3220
51. V. Gabillon, M. Ghavamzadeh, A. Lazaric, and S. Bubeck. Multi-bandit best arm identification. in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 2222-2230
52. J. R. Gardner, M. J. Kusner, Z. Xu, K. Q. Weinberger, and J. P. Cunningham. Bayesian optimization with inequality constraints. in Proc. Int. Conf. Mach. Learn., 2014, pp. 937-945
53. R. Garnett, M. A. Osborne, and P. Hennig. Active learning of linear embeddings for Gaussian processes. in Proc. Conf. Uncertainty Artif. Intell., 2014, pp. 24-33
54. R. Garnett, M. A. Osborne, S. Reece, A. Rogers, and S. J. Roberts. Sequential Bayesian prediction in the presence of changepoints and faults. Comput. J., vol. 53, no. 9, pp. 1430-1446, 2010
55. R. Garnett, M. A. Osborne, and S. J. Roberts. Bayesian optimization for sensor set selection. in Proc. ACM/IEEE Int. Conf. Inf. Process. Sensor Netw., 2010, pp. 209-219
56. M. A. Gelbart, J. Snoek, and R. P. Adams. Bayesian optimization with unknown constraints. in Proc. Conf. Uncertainty Artif. Intell., 2014, pp. 250-259
57. D. Ginsbourger, R. Le Riche, and L. Carraro. Kriging is well-suited to parallelize optimization. in Proc. Comput. Intell. Expensive Optim. Problems, 2010, pp. 131-162
58. D. Ginsbourger and R. L. Riche. Dealing with asynchronicity in parallel Gaussian process based global optimization. 2010. [Online]. Available: http://hal.archivesouvertes. fr/hal-00507632
59. J. C. Gittins. Bandit processes and dynamic allocation indices J. Roy. Stat. Soc. B, Methodol., vol. 2, pp. 148-177, 1979
60. P. Goovaerts, Geostatistics for Natural Resources Evaluation. Oxford, U.K.: Oxford Univ. Press, 1997
61. R. B. Gramacy., et al. Modeling an augmented Lagrangian for improved blackbox constrained optimization. 2014 [Online]. Available: arXiv: 1403.4890
62. R. B. Gramacy and H. K. Lee. Optimization under unknown constraints. 2010. [Online]. Available: arXiv: 1004.4027
63. R. B. Gramacy, H. K. H. Lee, and W. G. Macready. Parameter space exploration with Gaussian process trees. in Proc. Int. Conf. Mach. Learn., 2004, pp. 45-52
64. S. Grunewalder, J. Audibert, M. Opper, and J. Shawe-Taylor. Regret bounds for Gaussian process bandit problems. in Proc. 13th Int. Conf. Artif. Intell. Stat., 2010, pp. 273-280
65. F. Hamze, Z. Wang, and N. De Freitas. Self-Avoiding random dynamics on integer complex systems. ACM Trans. Model. Comput. Simul., vol. 23, no. 1, p. 9, 2013
66. N. Hansen and A. Ostermeier. Completely derandomized self-Adaptation in evolution strategies. Evol. Comput, vol. 9, no. 2, pp. 159-195, 2001
67. P. Hennig and C. Schuler. Entropy search for information-efficient global optimization. J. Mach. Learn. Res., vol. 13, no. 1, pp. 1809-1837, 2012
68. J. M. Hernández-Lobato, M. A. Gelbart, M. W. Hoffman, R. P. Adams, and Z. Ghahramani. Predictive entropy search for Bayesian optimization with unknown constraints. in Proc. Int. Conf. Mach. Learn., 2015, pp. 1699-1707
69. J. M. Hernández-Lobato, M. W. Hoffman, and Z. Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 918-926
70. D. Higdon, J. Swall, and J. Kern. Non-stationary spatial modeling. Bayesian Stat., vol. 6, 1998, pp. 761-768
71. G. E. Hinton and R. Salakhutdinov. Using deep belief nets to learn covariance kernels for Gaussian processes. in Proc. Adv. Neural Inf. Process. Syst., 2008, pp. 1249-1256
72. M. Hoffman, B. Shahriari, and N. De Freitas. On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning. in Proc. 17th Int. Conf. Artif. Intell. Stat., 2014, pp. 365-374
73. M. W. Hoffman, E. Brochu, and N. De Freitas. Portfolio allocation for Bayesian optimization. in Proc. Conf. Uncertainty Artif. Intell., 2011, pp. 327-336
74. H. H. Hoos. Programming by optimization. Commun. ACM, vol. 55, no. 2, pp. 70-80, 2012
75. D. Huang, T. Allen, W. Notz, and N. Zeng. Global optimization of stochastic black-box systems via sequential Kriging meta-models. J. Global Optim., vol. 34, no. 3, pp. 441-466, 2006
76. F. Hutter. Automated configuration of algorithms for solving hard computational problems. Ph.D. dissertation, Univ. British Columbia, Vancouver, BC, Canada, 2009
77. F. Hutter, H. Hoos, and K. Leyton-Brown. Identifying key algorithm parameters and instance features using forward selection. in Learning and Intelligent Optimization, vol. 7997, Berlin, Germany: Springer-Verlag, 2013, pp. 364-381
78. F. Hutter, H. H. Hoos, and K. Leyton-Brown. Automated configuration of mixed integer programming solvers. in Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, Berlin, Germany: Springer-Verlag, 2010, pp. 186-202
79. F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. Learning and Intelligent Optimization, Berlin, Germany: Springer-Verlag, 2011, pp. 507-523
80. F. Hutter, H. H. Hoos, and K. Leyton-Brown. Parallel algorithm configuration. Learning and Intelligent Optimization, Berlin, Germany: Springer-Verlag, 2012, pp. 55-70
81. D. Jones. A taxonomy of global optimization methods based on response surfaces. J. Global Optim., vol. 21, no. 4, pp. 345-383, 2001
82. D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black-box functions. J. Global Optim., vol. 13, no. 4, pp. 455-492, 1998
83. D. R. Jones, C. D. Perttunen, and B. E. Stuckman. Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl., vol. 79, no. 1, pp. 157-181, 1993
84. E. Kaufmann, O. Cappé, and A. Garivier. On Bayesian upper confidence bounds for bandit problems. in Proc. Int. Conf. Artif. Intell. Stat., 2012, pp. 592-600
85. E. Kaufmann, N. Korda, and R. Munos. Thompson sampling: An asymptotically optimal finite-Time analysis. in Algorithmic Learning Theory, vol. 7568, Berlin, Germany: Springer-Verlag, 2012, pp. 199-213
86. K. Kersting, C. Plagemann, P. Pfaff, and W. Burgard. Most likely heteroscedastic Gaussian process regression. in Proc. Int. Conf. Mach. Learn., 2007, pp. 393-400
87. L. Kocsis and C. Szepesvári. Bandit based Monte-Carlo planning. in Proc. Eur. Conf. Mach. Learn., 2006, pp. 282-293
88. R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne. Controlled experiments on the web: Survey and practical guide. Data Mining Knowl. Disc., vol. 18, no. 1, pp. 140-181, 2009
89. A. Krause and C. S. Ong. Contextual Gaussian process bandit optimization. in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 2447-2455
90. D. G. Krige. A statistical approach to some basic mine valuation problems on the witwatersrand. in J. Chem. Metallurgical Mining Soc. South Africa, vol. 94, no. 3, 1951, pp. 95-111
91. H. J. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Fluids Eng., vol. 86, no. 1, pp. 97-106, 1964
92. T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Adv. Appl. Math., vol. 6, no. 1, pp. 4-22, 1985
93. M. Lázaro-Gredilla and A. R. Figueiras-Vidal. Marginalized neural network mixtures for large-scale regression. IEEE Trans. Neural Netw., vol. 21, no. 8, pp. 1345-1351, Aug. 2010
94. M. Lázaro-Gredilla, J. Quinnonero-Candela, C. E. Rasmussen, and A. R. Figueiras-Vidal. Sparse spectrum Gaussian process regression. J. Mach. Learn. Res., vol. 11, pp. 1865-1881, 2010
95. Q. V. Le, A. J. Smola, and S. Canu. Heteroscedastic Gaussian process regression. in Proc. Int. Conf. Mach. Learn., 2005, pp. 489-496
96. K. Leyton-Brown, E. Nudelman, and Y. Shoham. Learning the empirical hardness of optimization problems: The case of combinatorial auctions. in Principles and Practice of Constraint Programming, ser. Lecture Notes in Computer Science, Berlin, Germany: Springer-Verlag, 2002, pp. 556-572
97. L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. in Proc. World Wide Web, 2010, pp. 661-670
98. D. V. Lindley. On a measure of the information provided by an experiment. Ann. Math. Stat., vol. 27, no. 4, pp. 986-1005, 1956
99. D. Lizotte. Practical Bayesian optimization. Ph.D. dissertation, Univ. Alberta, Edmonton, AB, Canada, 2008
100. D. Lizotte, R. Greiner, and D. Schuurmans. An experimental methodology for response surface optimization methods. J. Global Optim., vol. 53, pp. 1-38, 2011
101. D. Lizotte, T. Wang, M. Bowling, and D. Schuurmans. Automatic gait optimization with Gaussian process regression. in Proc. Int. Joint Conf. Artif. Intell., 2007, pp. 944-949
102. M. Locatelli. Bayesian algorithms for one-dimensional global optimization. J. Global Optim., vol. 10, pp. 57-76, 1997
103. M. Lzaro-gredilla and M. K. Titsias. Variational heteroscedastic Gaussian process regression. in Proc. Int. Conf. Mach. Learn., 2011, pp. 841-848, ACM
104. O. Madani, D. Lizotte, and R. Greiner. Active model selection. in Proc. Conf. Uncertainty Artif. Intell., 2004, pp. 357-365
105. N. Mahendran, Z. Wang, F. Hamze, and N. De Freitas. Adaptive MCMC with Bayesian optimization. J. Mach. Learn. Res., vol. 22, pp. 751-760, 2012
106. R. Marchant and F. Ramos. Bayesian optimisation for intelligent environmental monitoring. in NIPS Workshop Bayesian Optim. Decision Making, 2012
107. O. Maron and A. W. Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. Robot. Inst., vol. 6, pp. 59-66, 1993
108. R.Martinez-Cantin, N. De Freitas, E. Brochu, J. Castellanos, and A. Doucet. A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot. Autonom. Robots, vol. 27, no. 2, pp. 93-103, 2009
109. J. Martinez, J. J. Little, and N. De Freitas. Bayesian optimization with an empirical hardness model for approximate nearest neighbour search. in Proc. IEEE Winter Conf. Appl. Comput. Vis., 2014, pp. 588-595
110. R.Martinez-Cantin, N. De Freitas, A. Doucet, and J. A. Castellanos. Active policy learning for robot planning and exploration under uncertainty. in Proc. Robot. Sci. Syst., pp. 321-328, 2007
111. G. Matheron. The theory of regionalized variables and its applications. in Cahier du Centre De Morphologie Mathematique, Ecoles des Mines, 1971
112. B. C. May, N. Korda, A. Lee, and D. S. Leslie. Optimistic Bayesian sampling in contextual bandit problems. Stat. Group, Schl. Math., Univ. Bristol, Bristol, U.K., Tech. Rep. 11: 01, 2011
113. V. Mnih, C. Szepesvári, and J.-Y. Audibert. Empirical Bernstein stopping. in Proc. Int. Conf. Mach. Learn., 2008, pp. 672-679
114. J. Močkus. Application of Bayesian approach to numerical methods of global and stochastic optimization. J. Global Optim., vol. 4, no. 4, pp. 347-365, 1994
115. J. Močkus, V. Tiesis, and A. Zilinskas. The application of Bayesian methods for seeking the extremum. in Toward Global Optimization, vol. 2, L. Dixon and G. Szego, Eds. Amsterdam, The Netherlands: Elsevier, 1978
116. R. Munos. Optimistic optimization of a deterministic function without the knowledge of its smoothness. in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 783-791
117. R. Munos. From bandits to Monte-Carlo tree search: The optimistic principle applied to optimization and planning. INRIA Lille, France, Tech. Rep. hal-00747575, 2014
118. R. M. Neal. Bayesian learning for neural networks. Ph.D. dissertation, Univ. Toronto, Toronto, ON, Canada, 1995
119. J. Nelder and R. Wedderburn. Generalized linear models. J. Roy. Stat. Soc. A, vol. 135, no. 3, pp. 370-384, 1972
120. M. A. Osborne, R. Garnett, and S. J. Roberts. Gaussian processes for global optimisation Learning and Intelligent Optimization, Berlin, Germany: Springer-Verlag, pp. 1-15, 2009
121. C. Paciorek and M. Schervish. Nonstationary covariance functions for Gaussian process regression. in Proc. Adv. Neural Inf. Process. Syst., 2004, vol. 16, pp. 273-280
122. V. Picheny and D. Ginsbourger. A nonstationary space-Time Gaussian process model for partially converged simulations. SIAM/ASA J. Uncertainty Quantif., vol. 1, no. 1, pp. 57-78, 2013
123. J. C. Pinheiro and D. M. Bates. Unconstrained parametrizations for variance-covariance matrices. Stat. Comput., vol. 6, no. 3, pp. 289-296, 1996
124. J. Qui nonero-Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. J. Mach. Learn. Res., vol. 6, pp. 1939-1959, 2005
125. A. Rahimi and B. Recht. Random features for large-scale kernel machines. in Proc. Adv. Neural Inf. Process. Syst., 2007, pp. 1177-1184
126. C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, MA, USA: MIT Press, 2006
127. D. Russo and B. Van Roy. Learning to optimize via posterior sampling. Math. Oper. Res., vol. 39, no. 4, pp. 1221-1243, 2014
128. J. Sacks, W. J. Welch, T. J. Welch, and H. P. Wynn. Design and analysis of computer experiments. Stat. Sci., vol. 4, no. 4, pp. 409-423, 1989
129. P. D. Sampson and P. Guttorp. Nonparametric estimation of nonstationary spatial covariance structure. J. Amer. Stat. Assoc., vol. 87, no. 417, pp. 108-119, 1992
130. T. J. Santner, B. Williams, and W. Notz, The Design and Analysis of Computer Experiments. New York, NY, USA: Springer-Verlag, 2003
131. M. J. Sasena. Flexibility and efficiency enhancement for constrained global design optimization with Kriging approximations. Ph.D. dissertation, Univ. Michigan, Ann Arbor, MI, USA, 2002
132. A. M. Schmidt and A. O'Hagan. Bayesian inference for nonstationary spatial covariance structures via spatial deformations. J. Roy. Stat. Soc. B, vol. 65, pp. 743-758, 2003
133. M. Schonlau. Computer experiments and global optimization. Ph.D. dissertation, Univ. Waterloo, Waterloo, ON, Canada, 1997
134. M. Schonlau, W. J. Welch, and D. R. Jones. Global versus local search in constrained optimization of computer models. Lecture Notes-Monograph Series, vol. 34, pp. 11-25, 1998
135. S. L. Scott. A modern Bayesian look at the multi-Armed bandit. Appl. Stochastic Models Business Ind., vol. 26, no. 6, pp. 639-658, 2010
136. M. Seeger, Y.-W. Teh, and M. I. Jordan. Semiparametric latent factor models. in Proc. Int. Conf. Artif. Intell. Stat., 2005, pp. 333-340
137. M. Seeger, C. Williams, and N. Lawrence. Fast forward selection to speed up sparse Gaussian process regression. in Proc. Artif. Intell. Stat. 9, 2003, pp. 1-8
138. A. Shah, A. G. Wilson, and Z. Ghahramani. Student-T processes as alternatives to Gaussian processes. in Proc. Int. Conf. Artif. Intell. Stat., 2014, pp. 877-885
139. B. Shahriari, Z. Wang, M. W. Hoffman, A. Bouchard-Coˆté, and N. De Freitas. An entropy search portfolio. in Proc. NIPS Workshop Bayesian Optim., 2014
140. E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. in Proc. Adv. Neural Inf. Process. Syst., 2005, pp. 1257-1264
141. E. Snelson, C. E. Rasmussen, and Z. Ghahramani. Warped Gaussian processes. in Proc. Adv. Neural Inf. Process. Syst., 2003, pp. 337-344
142. J. Snoek. Bayesian optimization and semiparametric models with applications to assistive technology. Ph.D. dissertation, Univ. Toronto, Toronto, ON, Canada, 2013
143. J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 2951-2959
144. J. Snoek., et al. Scalable Bayesian optimization using deep neural networks. in Proc. Int. Conf. Mach. Learn., 2015, pp. 2171-2180
145. J. Snoek, K. Swersky, R. S. Zemel, and R. P. Adams. Input warping for Bayesian optimization of non-stationary functions. in Proc. Int. Conf. Mach. Learn., 2014, pp. 1674-1682
146. N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. in Proc. Int. Conf. Mach. Learn., 2010, pp. 1015-1022
147. K. Swersky, D. Duvenaud, J. Snoek, F. Hutter, and M. A. Osborne. Raiders of the lost architecture: Kernels for Bayesian optimization in conditional parameter spaces. 2014. [Online]. Available: arXiv: 1409.4011
148. K. Swersky, J. Snoek, and R. P. Adams. Multi-Task Bayesian optimization. in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 2004-2012
149. K. Swersky, J. Snoek, and R. P. Adams. Freeze-Thaw Bayesian optimization. 2014. [Online]. Available: arXiv: 1406.3896
150. W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, vol. 25, no. 3/4, pp. 285-294, 1933
151. C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. in Proc. Knowl. Disc. Data Mining, 2013, pp. 847-855
152. M. K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. in Proc. Int. Conf. Artif. Intell. Stat., 2009, pp. 567-574
153. H. P. Vanchinathan, I. Nikolic, F. De Bona, and A. Krause. Explore-exploit in top-N recommender systems via Gaussian processes. in Proc. 8th ACM Conf. Recommender Syst., 2014, pp. 225-232
154. E. Vazquez and J. Bect. Convergence properties of the expected improvement algorithm with fixed mean and covariance functions. J. Stat. Planning Inference, vol. 140, no. 11, pp. 3088-3095, 2010
155. J. Villemonteix, E. Vazquez, and E. Walter. An informational approach to the global optimization of expensive-To-evaluate functions. J. Global Optim., vol. 44, no. 4, pp. 509-534, 2009
156. Z. Wang and N. De Freitas. Theoretical analysis of Bayesian optimisation with unknown Gaussian process hyper-parameters. 2014. [Online]. Available: arXiv: 1406.7758
157. Z. Wang, B. Shakibi, L. Jin, and N. De Freitas. Bayesian multi-scale optimistic optimization. in Proc. Int. Conf. Artif. Intell. Stat., 2014, pp. 1005-1014
158. Z. Wang, M. Zoghi, D. Matheson, F. Hutter, and N. De Freitas. Bayesian optimization in high dimensions via random embeddings. in Proc. Int. Joint Conf. Artif. Intell., 2013, pp. 1778-1784
159. B. J. Williams, T. J. Santner, and W. I. Notz. Sequential design of computer experiments to minimize integrated response functions. Statistica Sinica, vol. 10, pp. 1133-1152, 2000
160. D. Yogatama and G. Mann. Efficient transfer learning method for automatic hyperparameter tuning. in Proc. Int. Conf. Artif. Intell. Stat., 2014, pp. 1077-1085
161. D. Yogatama and N. A. Smith. Bayesian optimization of text representations. 2015. [Online]. Available: arXiv: 1503.00693
162. Y. Zhang, K. Sohn, R. Villegas, G. Pan, and H. Lee. Improving object detection with deep convolutional networks via Bayesian optimization and structured prediction. in Proc. IEEE Comput. Vis. Pattern Recognit. Conf., 2015, pp. 249-258
163. A. Zilinskas and J. Zilinskas. Global optimization based on a statistical model and simplical partitioning. Comput. Math. Appl., vol. 44, pp. 957-967, 2002