Hierarchical knowledge gradient for sequential sampling

Journal of Machine Learning Research

Volumn 12, Issue , 2011, Pages 2931-2974

  Mes, Martijn R K ^a   Powell, Warren B ^b   Frazier, Peter I ^c  

^a UNIVERSITY OF TWENTE   (Netherlands)

^b Princeton University   (United States)

^c Cornell University   (United States)

Author keywords

Adaptive learning; Bayesian statistics; Hierarchical statistics; Ranking and selection; Sequential experimental design

Indexed keywords

ADAPTIVE LEARNING; BAYESIAN STATISTICS; HIERARCHICAL STATISTICS; RANKING AND SELECTION; SEQUENTIAL EXPERIMENTAL DESIGN;

GLOBAL OPTIMIZATION; OPTIMIZATION;

SAMPLING;

EID: 80555145311     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (56)

1. Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Proceedings of the 21st Annual Conference on Learning Theory (COLT), pages 263-274, 2008.
2. Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002. (Pubitemid 34126111)
3. Peter L. Bartlett, Varsha Dani, Thomas P. Hayes, Sham Kakade, Alexander Rakhlin, and Ambuj Tewari. High-probability regret bounds for bandit online linear optimization. In Proceedings of the 21st Annual Conference on Learning Theory (COLT), pages 335-342, 2008.
4. Russell R. Barton and Martin Meckesheimer. Metamodel-based simulation optimization. In Shane G. Henderson and Barry L. Nelson, editors, Simulation, volume 13 of Handbooks in Operations Research and Management Science, pages 535-574. Elsevier, 2006.
5. Robert E. Bechhofer. A single-sample multiple decision procedure for ranking means of normal populations with known variances. The Annals of Mathematical Statistics, 25(1):16-39, 1954.
6. Robert E. Bechhofer, Thomas J. Santner, and David M. Goldsman. Design and Analysis of Experiments for Statistical Selection, Screening and Multiple Comparisons. John Wiley & Sons, New York, NY, 1995.
7. Dimitri P. Bertsekas and David A. Castanon. Adaptive aggregation methods for infinite horizon dynamic programming. IEEE Transactions on Automatic Control, 34(6):589-598, 1989.
8. Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996.
9. Ronen I. Brafman and Moshe Tennenholtz. R-MAX - a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3:213-231, 2003.
10. Franklin H. Branin. Widely convergent method for finding multiple solutions of simultaneous nonlinear equations. IBM Journal of Reseach and Development, 16(5):504-522, 1972.
11. Erik Brochu, Mike Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Technical Report TR-2009-023, Department of Computer Science, University of British Columbia, 2009.
12. Sébastien Bubeck, Réemi Munos, Gilles Stoltz, and Csaba Szepesvári. Online optimization in X-armed bandits. In Advances in Neural Information Processing Systems, pages 201-208, 2009a.
13. Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In Proceedings of the 20th International Conference on Algorithmic Learning Theory, pages 23-37, 2009b.
14. Chun-Hung Chen, Hsiao-Chang Chen, and Liyi Dai. A gradient approach for smartly allocating computing budget for discrete event simulation. In Proceedings of the 28th Conference on Winter Simulation, pages 398-405, 1996.
15. Stephen E. Chick and Koichiro Inoue. New two-stage and sequential procedures for selecting the best simulated system. Operations Research, 49(5):732-743, 2001. (Pubitemid 34011324)
16. Stephen E. Chick, Jurgen Branke, and Christian Schmidt. Sequential sampling to myopically maximize the expected value of information. INFORMS Journal on Computing, 22(1):71-80, 2010.
17. Eyal Even-Dar, Shie Mannor, and Yishay Mansour. PAC bounds for multi-armed bandit and Markov decision processes. In Proceedings of the 15th Annual Conference on Computational Learning Theory (COLT), pages 193-209, 2002.
18. Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and stopping conditions for reinforcement learning. In Proceedings of the 20th International Conference on Machine Learning, pages 162-169, 2003.
19. Abraham D. Flaxman, Adam Tauman Kalai, and H. Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the Sixteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '05, pages 385-394, 2005. (Pubitemid 40851394)
20. Peter I. Frazier, Warren B. Powell, and Savas Dayanik. A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization, 47(5):2410-2439, 2008.
21. Peter I. Frazier, Warren B. Powell, and Savas Dayanik. The knowledge-gradient policy for correlated normal beliefs. INFORMS Journal on Computing, 21(4):599-613, 2009.
22. Abraham George, Warren B. Powell, and Sanjeev R. Kulkarni. Value function approximation using multiple aggregation for multiattribute resource management. Journal of Machine Learning Research, 9:2079-2111, 2008.
23. Mark N. Gibbs. Bayesian Gaussian Processes for Regression and Classification. PhD thesis, University of Cambridge, 1997.
24. Steffen Grünewälder, Jean-Yves Audibert, Manfred Opper, and John Shawe-Taylor. Regret bounds for gaussian process bandit problems. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
25. Shanti S. Gupta and Klaus J. Miescke. Bayesian look ahead one-stage sampling allocations for selection of the best population. Journal of Statistical Planning and Inference, 54(2):229-244, 1996. (Pubitemid 126161097)
26. Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The Elements of Statistical Learning. Springer series in Statistics, New York, NY, 2001.
27. Donghai He, Stephen E. Chick, and Chun-Hung Chen. Opportunity cost and OCBA selection procedures in ordinal optimization for a fixed number of alternative systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 37(5):951-961, 2007. (Pubitemid 47323564)
28. Deng Huang, Theodore T. Allen, William I. Notz, and Ning Zheng. Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal of Global Optimization, 34(3):441-466, 2006. (Pubitemid 43352177)
29. Frank Hutter. Automated Configuration of Algorithms for Solving Hard Computational Problems. PhD thesis, University of British Columbia, 2009.
30. Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455-492, 1998. (Pubitemid 128507405)
31. Leslie P. Kaelbling. Learning In Embedded Systems. MIT Press, Cambridge, MA, 1993.
32. Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49(2-3):209-232, 2002. (Pubitemid 34325687)
33. Seong-Hee Kim and Barry L. Nelson. Handbook in Operations Research and Management Science: Simulation, chapter Selecting the best system. Elsevier, Amsterdam, 2006.
34. Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in metric spaces. In Proceedings of the 40th Annual ACM symposium on Theory of Computing, pages 681-690, 2008.
35. Robert D. Kleinberg. Online Decision Problems With Large Strategy Sets. PhD thesis, MIT, 2005.
36. Harold J. Kushner. A new method of locating the maximum of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, 86:97-106, 1964.
37. Tze L. Lai. Adaptive treatment allocation and the multi-armed bandit problem. The Annals of Statistics, 15(3):1091-1114, 1987.
38. Michael LeBlanc and Robert Tibshirani. Combining estimates in regression and classification. Journal of the American Statistical Association, 91(436):1641-1650, 1996.
39. Nick Littlestone. From on-line to batch learning. In Proceedings of the Second Annual Workshop on Computational learning theory, pages 269-284, 1989.
40. Daniel J. Lizotte. Practical Bayesian Optimization. PhD thesis, University of Alberta, 2008.
41. Omid Madani, Daniel J. Lizotte, and Russell Greiner. The budgeted multi-armed bandit problem. In Proceedings of the 17th Annual Conference on Computational Learning Theory (COLT), pages 643-645, 2004.
42. Volodymyr Mnih, Csaba Szepesvári, and Jean-Yves Audibert. Empirical Bernstein stopping. In Proceedings of the 25th International Conference on Machine Learning, pages 672-679, 2008.
43. Jonas Mockus. On Bayesian methods for seeking the extremum. In G. Marchuk, editor, Optimization Techniques IFIP Technical Conference Novosibirsk, July 17, 1974, volume 27 of Lecture Notes in Computer Science, pages 400-404. Springer Berlin/Heidelberg, 1975.
44. Warren B. Powell and Peter I. Frazier. Optimal learning. In TutORials in Operations Research, pages 213-246. INFORMS, 2008.
45. Carl E. Rasmussen and Christopher Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
46. Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22(3):400-407, 1951.
47. David F. Rogers, Robert D. Plante, Richard T. Wong, and James R. Evans. Aggregation and disaggregation techniques and methodology in optimization. Operations Research, 39(4):553-582, 1991.
48. Michael J. Sasena. Flexibility and Efficiency Enhancements for Constrained Global Design Optimization with Kriging Approximations. PhD thesis, University of Michigan, 2002.
49. Shai Shalev-Shwartz. Online learning: Theory, algorithms, and applications. PhD thesis, The Hebrew University of Jerusalem, 2007.
50. Hugo P. Simao, Jeff Day, Abraham P. George, Ted Gifford, John Nienow, and Warren B. Powell. An approximate dynamic programming algorithm for large-scale fleet management: A case application. Transportation Science, 43(2):178-197, 2009.
51. Tom A. B. Snijders and Roel J. Bosker. Multilevel Analysis: An Introduction To Basic And Advanced Multilevel Modeling. Sage Publications Ltd, 1999.
52. James C. Spall. Introduction to Stochastic Search and Optimization. Wiley-Interscience, Hoboken, NJ, 2003.
53. Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings International Conference on Machine Learning (ICML), 2010.
54. Emmanuel Vazquez and Julien Bect. Convergence properties of the expected improvement algorithm with fixed mean and covariance functions. Journal of Statistical Planning and Inference, 140(11):3088-3095, 2010.
55. Julien Villemonteix, Emmanuel Vazquez, and Eric Walter. An informational approach to the global optimization of expensive-to-evaluate functions. Journal of Global Optimization, 44(4):509-534, 2009.
56. Yuhong Yang. Adaptive regression by mixing. Journal of American Statistical Association, 96(454):574-588, 2001.