The knowledge gradient algorithm for a general class of online learning problems

Operations Research

Volumn 60, Issue 1, 2012, Pages 180-195

  Ryzhov, Ilya O ^a   Powell, Warren B ^b   Frazier, Peter I ^c  

^a UNIVERSITY OF MARYLAND   (United States)

^b Princeton University   (United States)

^c Cornell University   (United States)

Author keywords

Gittins index; Index policy; Knowledge gradient; Multiarmed bandit; Online learning; Optimal learning

Indexed keywords

GITTINS INDEX; INDEX POLICIES; KNOWLEDGE GRADIENT; MULTI ARMED BANDIT; ONLINE LEARNING; OPTIMAL LEARNING;

OPTIMIZATION;

E-LEARNING;

EID: 84859621831     PISSN: 0030364X     EISSN: 15265463     Source Type: Journal    
DOI: 10.1287/opre.1110.0999     Document Type: Article

References (40)

1. Auer, P., N. Cesa-Bianchi, P. Fischer. 2002. Finite-time analysis of the multiarmed bandit problem. Machine Learn. 47(2-3) 235-256. (Pubitemid 34126111)
2. Berry, D. A., L. M. Pearson. 1985. Optimal designs for clinical trials with dichotomous responses. Statist. Medicine 4(4) 497-508. (Pubitemid 16176681)
3. Bertsimas, D., J. Niño-Mora. 2000. Restless bandits, linear programming relaxations, and a primal-dual index heuristic. Oper. Res. 48(1) 80-90.
4. Brezzi, M., T. L. Lai. 2000. Incomplete learning from endogenous data in dynamic allocation. Econometrica 68(6) 1511-1516.
5. Brezzi, M., T. L. Lai. 2002. Optimal learning and experimentation in bandit problems. J. Econom. Dynam. Control 27(1) 87-108.
6. Chick, S. E., N. Gans. 2009. Economic analysis of simulation selection problems. Management Sci. 55(3) 421-437.
7. Chick, S. E., J. Branke, C. Schmidt. 2010. Sequential sampling to myopi-cally maximize the expected value of information. INFORMS J. Comput. 22(1) 71-80.
8. DeGroot, M. H. 1970. Optimal Statistical Decisions. John Wiley & Sons, Hoboken, NJ.
9. Duff, M. 1995. Q-learning for bandit problems. Proc. 12th Internat. Conf. Machine Learn., Morgan Kaufmann, Tahoe City, CA, 209-217.
10. Feldman, D. 1962. Contributions to the two-armed bandit problem. Ann. Math. Statist. 33(3) 847-856.
11. Frazier, P. I., W. B. Powell, S. Dayanik. 2008. A knowledge gradient policy for sequential information collection. SIAM J. Control Optim. 47(5) 2410-2439.
12. Frazier, P. I., W. B. Powell, S. Dayanik. 2009. The knowledge-gradient policy for correlated normal rewards. INFORMS J. Comput. 21(4) 599-613.
13. Ginebra, J., M. K. Clayton. 1995. Response surface bandits. J. Royal Statist. Soc. B57(4) 771-784.
14. Gittins, J. C. 1989. Multi-Armed Bandit Allocation Indices. John Wiley & Sons, New York.
15. Gittins, J. C., D. M. Jones. 1974. A dynamic allocation index for the sequential design of experiments. J. Gani, ed. Progress in Statistics. North-Holland, Amsterdam, 244-266.
16. Gittins, J. C., D. M. Jones. 1979. A dynamic allocation index for the discounted multiarmed bandit problem. Biometrika 66(3) 561-565. (Pubitemid 10218405)
17. Goel, A., S. Khanna, B. Null. 2009. The ratio index for budgeted learning, with applications. Proc. 19th Annual ACM-SIAM Sympos. Discrete Algorithms, SIAM, Philadelphia, 18-27.
18. Gupta, S. S., K. J. Miescke. 1994. Bayesian look ahead one stage sampling allocations for selecting the largest normal mean. Statist. Papers 35(1) 169-177.
19. Gupta, S. S., K. J. Miescke. 1996. Bayesian look ahead one-stage sampling allocations for selection of the best population. J. Statist. Planning Inference 54(2) 229-244. (Pubitemid 126161097)
20. Kaelbling, L. P. 1993. Learning in Embedded Systems. MIT Press, Cambridge, MA.
21. Katehakis, M. N., A. F. Veinott. 1987. The multi-armed bandit problem: Decomposition and computation. Math. Oper. Res. 12(2) 262-268.
22. Keener, R. 1985. Further contributions to the "two-armed bandit" problem. Ann. Statist. 13(1) 418-422.
23. Lai, T. L. 1987. Adaptive treatment allocation and the multi-armed bandit problem. Ann. Statist. 15(3) 1091-1114.
24. Lai, T. L., H. Robbins. 1985. Asymptotically efficient adaptive allocation rules. Adv. Appl. Math. 6(1) 4-22.
25. Mahajan, A., D. Teneketzis. 2008. Multi-armed bandit problems. A. O. Hero, D. A. Castanon, D. Cochran, K. Kastella, eds. Foundations and Applications of Sensor Management. Springer, New York, 121-152.
26. Mersereau, A. J., P. Rusmevichientong, J. N. Tsitsiklis. 2008. A structured multiarmed bandit problem and the greedy policy. Proc. 47th IEEE Conf. Decision and Control, IEEE, Piscataway, NJ, 4945-4950.
27. Mersereau, A. J., P. Rusmevichientong, J. N. Tsitsiklis. 2009. A structured multiarmed bandit problem and the greedy policy. IEEE Trans. Automatic Control 54(12) 2787-2802.
28. Pandey, S., D. Chakrabarti, D. Agarwal. 2007. Multi-armed bandit problems with dependent arms. Proc. 24th Internat. Conf. Machine Learn., ACM International Proceedings Series, New York, 721-728.
29. Powell, W. B. 2007. Approximate Dynamic Programming: Solving the Curses of Dimensionality. John Wiley & Sons, New York.
30. Ryzhov, I. O., W. B. Powell. 2009a. A Monte Carlo knowledge gradient method for learning abatement potential of emissions reduction technologies. M. D. Rosetti, R. R. Hill, B. Johansson, A. Dunkin, R. G. Ingalls, eds. Proc. 2009 Winter Simulation Conf., IEEE, Piscataway, NJ, 1492-1502.
31. Ryzhov, I. O., W. B. Powell. 2009b. The knowledge gradient algorithm for online subset selection. Proc. 2009 IEEE Sympos. Adaptive Dynam. Programming and Reinforcement Learn., IEEE, Piscataway, NJ, 137-144.
32. Ryzhov, I. O., W. B. Powell. 2011. Information collection on a graph. Oper. Res. 59(1) 188-201.
33. Steele, J. M. 2000. Stochastic Calculus and Financial Applications. Springer, New York.
34. Sutton, R. S., A. G. Barto. 1998. Reinforcement Learning. MIT Press, Cambridge, MA.
35. Tesauro, G., G. R. Galperin. 1996. On-line policy improvement using Monte Carlo search. M. C. Mozer, M. I. Jordan, T. Pesche, eds. Advances in Neural Information Processing Systems, Vol. 9. MIT Press, Cambridge, MA, 1068-1074.
36. Tewari, A., P. L. Bartlett. 2007. Optimistic linear programming gives logarithmic regret for irreducible MDPs. J. C. Platt, D. Koller, Y. Singer, S. Roweis, eds. Advances in Neural Information Processing Systems, Vol. 20. MIT Press, Cambridge, MA, 1505-1512.
37. Vermorel, J., M. Mohri. 2005. Multi-armed bandit algorithms and empirical evaluation. Proc. 16th Eur. Conf. Machine Learn., SpringerVerlag, Berlin, 437-448.
38. Washburn, R. 2008. Applications of multi-armed bandits to sensor management. A. O. Hero, D. A. Castanon, D. Cochran, K. Kastella, eds. Foundations and Applications of Sensor Management. Springer, New York, 153-176.
39. Whittle, P. 1980. Multi-armed bandits and the Gittins index. J. Royal Statist. Soc. B42(2) 143-149.
40. Yao, Y. 2006. Some results on the Gittins index for a normal reward process. H. Ho, C. Ing, T. Lai, eds. Time Series and Related Topics: In Memory of Ching-Zong Wei. Institute of Mathematical Statistics, Beachwood, OH, 284-294.