Multi-Armed Bandit Allocation Indices: 2nd Edition

Multi-Armed Bandit Allocation Indices: 2nd Edition

Volumn , Issue , 2011, Pages

  Gittins, John ^a   Glazebrook, Kevin ^b   Weber, Richard ^c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b LANCASTER UNIVERSITY   (United Kingdom)

^c UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84891584370     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9780470980033     Document Type: Book

References (225)

1. Aalto, S., Ayesta, U. and Righter, R. (2009). On the Gittins index in the M/G/1 queue, Queueing Syst. 63, 437-58.
2. Agrawal, R., Hedge, M. and Teneketzis, D. (1988). Asymptotically efficient adaptive allocation rules for the multiarmed bandit problem with switching costs, IEEE T. Automat. Contr. 33, 899-906.
3. Ahlswede, R. and Wegener, I. (1987). Search Problems, John Wiley & Sons, Ltd.
4. Ahmad, S.H.A., Liu, M., Javidi, T., Zhao, Q. and Krishnamachari, B. (2009). Optimality of myopic sensing in multichannel opportunistic access, IEEE T. Inf. Theor. 55, 4040-50.
5. Alpern, S. and Gal, S. (2003). The Theory of Search Games and Rendezvous, Springer.
6. Amaral, J.A.F.P., do (1985). Aspects of Optimal Sequential Resource Allocation, D.Phil. thesis, Oxford.
7. Anantharam, V., Varaiya, P. and Walrand, J. (1987a). Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays -Part I: I.I.D. rewards, IEEE T. Automat. Contr. 32, 968-76.
8. Anantharam, V., Varaiya, P. and Walrand, J. (1987b). Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays -Part II: Markovian rewards, IEEE T. Automat. Contr. 32, 977-82.
9. Ansell, P.S., Glazebrook, K.D., Niño-Mora, J. and O'Keeffe, M. (2003). Whittle's index policy for a multi-class queueing system with convex holding costs, Math. Meth. Oper. Res. 57, 21-39.
10. Aviv, Y. and Kress, M. (1997). Evaluating the effectiveness of shoot-look-shoot tactics in the presence of incomplete damage information, Military Oper. Res. 3, 79-89.
11. Babaioff, M., Sharma, Y. and Slivkins, A. (2009). Characterizing truthful multi-armed bandit mechanisms: extended abstract, in EC '09: Proceedings of the 2009 ACM Conference Electronic Commerce, ACM, pages 79-88.
12. Baker, K.R. (1974). Introduction to Sequencing and Scheduling, John Wiley & Sons, Ltd.
13. Bank, P. and Küchler, C. (2007). On Gittins index theorem in continuous time, Stoc. Proc. Appl. 117, 1357-71.
14. Banks, J.S. and Sundaram, R.K. (1994). Switching costs and the Gittins index, Econometrica 62, 687-94.
15. Barkdoll, T., Gaver, D.P., Glazebrook, K.D., Jacobs, P.A. and Posadas, S. (2002). Suppression of enemy air defenses (SEAD) as an information duel, Naval Res. Logist. 49, 723-42.
16. Barra, J.-R. (1981). Mathematical Basis of Statistics, Academic Press.
17. Bassamboo, A., Harrison, J.M. and Zeevi, A. (2005). Dynamic routing and admission control in high-volume service systems: asymptotic analysis via multi-scale fluid limits, Queueing Syst. 51, 249-85.
18. Bather, J.A. (1962). Bayesian procedures for deciding the sign of a normal mean, Proc. Cambridge Phil. Soc. 58, 599-620.
19. Bather, J.A. (1970). Optimal stopping problems for Brownian motion, Adv. Appl. Prob. 2, 259-86.
20. Bather, J.A. (1983). Optimal stopping of Browian motion: a comparison technique, in D. Siegmund, H. Chernoff, J.S. Rustagi and M.H. Rizvi, editors, Recent Advances in Statistics, Academic Press, pages 19-50.
21. Bäuerle, N. and Stidham, S., Jr. (2001). Conservation laws for single-server fluid networks, Queueing Syst. 38, 185-94.
22. Beale, E.M.L. (1979). Contribution to the discussion of Gittins, J. R. Statist. Soc. B 41, 171-2.
23. Bellman, R.E. (1956). A problem in the sequential design of experiments, Sankhya A 30, 221-52.
24. Bellman, R.E. (1957). Dynamic Programming, Princeton University Press.
25. Ben-Israel, A. and Fl°am, S.D. (1990). A bisection/successive approximation method for computing Gittins indices, Math. Meth. Oper. Res. 34, 411-22.
26. Bensoussan, A. and Lions, J.L. (1978). Applications des Inequations Variationelles en Controle Stochastique, Dunod.
27. Bergemann, D. and Välimäki, J. (2001). Stationary multi-choice bandit problems, J. Econ. Dynam. Control 25, 158-94.
28. Bergemann, D. and Välimäki, J. (2008). Bandit problems, in S.N. Durlauf and L.E. Blume, editors, The New Palgrave Dictionary of Economics, Palgrave Macmillan, pages 336-40.
29. Bergman, S.W. (1981). Acceptance Sampling: The Buyer's Problem, Ph.D. thesis, Yale.
30. Bergman, S.W. and Gittins, J.C. (1985). Statistical Methods for Planning Pharmaceutical Research, Marcel Dekker.
31. Berninghaus, S. (1984). Das 'Multi-Armed-Bandit' Paradigms, Verlagsgruppe Athenaum.
32. Berry, D.A. and Fristedt, B. (1985). Bandit Problems, Chapman Hall.
33. Berry, D.A. and Kertz, R.P. (1991). Worth of perfect information in Bernoulli bandits, Adv. Appl. Prob. 23, 1-23.
34. Bertsekas, D.P. (1999). Nonlinear Programming, Athena Scientific.
35. Bertsimas, D. and Niño-Mora, J. (1993). Conservation laws, extended polymatroids and multi-armed bandit problems; a polyhedral approach to indexable systems, in G. Rinaldi and L. Wolsey, editors, Proceedings of the Third International Symposium on Integer Programming and Combinatorial Optimization (IPCO 93), Erice, Italy, CIACO, pages 355-84.
36. Bertsimas, D. and Niño-Mora, J. (1996). Conservation laws, extended polymatroids and multiarmed bandit problems; a polyhedral approach to indexable systems, Math. Oper. Res. 21, 257-306.
37. Bertsimas, D., Paschalidis, I. and Tsitsiklis, J.N. (1995). Branching bandits and Klimov's problem: achievable region and side constraints, IEEE T. Automat. Contr. 40, 2063-75.
38. Bertsimas, D.J. and Tsitsiklis, J.N. (1997). Introduction to Linear Optimization, Athena Scientific.
39. Bhattacharya, P.P., Georgiadis, L. and Tsoucas, P. (1992). Extended polymatroids: properties and optimization, in E. Balas, G. Cornnjelos and R. Kannan, editors, Proceedings of the 2nd Integer Programming and Combinatorial Optimization Conference, Pittsburgh, University Printing and Publications, Carnegie Mellon University, pages 298-315.
40. Bhulai, S. and Koole, G.M. (2000). On the value of learning for Bernoulli bandits with unknown parameters, IEEE T. Automat. Contr. 45, 213-40.
41. Blackwell, D. (1962). Discrete dynamic programming, Ann. Math. Stat. 39, 719-26.
42. Blackwell, D. (1965). Discounted dynamic programming, Ann. Math. Stat. 36, 226-35.
43. Bradt, R.N., Johnson, S.M. and Karlin, S. (1956). On sequential designs for maximizing the sum of n observations, Ann. Math. Stat. 27, 1060-74.
44. Cavallo, R. (2008). Efficiency and redistribution in dynamic mechanism design, in EC'08: Proceedings of the 2008 ACM conference on Electronic Commerce, ACM, pages 220-229.
45. Cavallo, R., Parkes, D.C. and Singh, S. (2006). Optimal coordinated planning amongst self-interested agents with private state, in R. Dechter and T. Richardson, editors, Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence (UAI'06), Cambridge, MA, AVAI Press, pages 55-62.
46. Chang, F. and Lai, T.L. (1987). Optimal stopping and dynamic allocation, Adv. Appl. Prob. 19, 829-53.
47. Chen, Y.R. and Katehakis, M.N. (1986). Linear programming for finite state multi-armed bandit problems, Math. Oper. Res., 11, 83.
48. Chernoff, H. (1961). Sequential tests for the mean of a normal distribution, in J. Neyman, editor, Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, University of California Press, pages 79-81.
49. Chernoff, H. (1968). Optimal stochastic control, Sankhya A 30, 221-52.
50. Chow, Y.S., Robbins, H. and Siegmund, D. (1971). Great Expectations: The Theory of Optimal Stopping, Houghton Mifflin.
51. Coffman, E.G., Jr. (1976). Computer and Job-Shop Scheduling Theory, John Wiley & Sons, Inc.
52. Coffman, E.G., Jr. and Mitrani, I. (1980). A characterization of waiting time performance realizable by single-server queues, Oper. Res. 28, 810-21.
53. Conway, R.W., Maxwell, W.L. and Miller, L.W. (1967). Theory of Scheduling, Addison-Wesley.
54. Cramér, H. (1946). Mathematical Methods of Statistics, Princeton University Press.
55. Dacre, M.J. and Glazebrook, K.D. (2002). The dependence of optimal returns from multiclass queueing systems on their customer base, Queueing Syst. 40, 93-115.
56. Dacre, M.J., Glazebrook, K.D. and Niño-Mora, J. (1999). The achievable region approach to the optimal control of stochastic systems, J. R. Statist. Soc. B, 747-91 (with discussion).
57. Dempster, M.A.H., Lenstra, J.K. and Rinnooy Kan, A.H.G., editors (1982). Deterministic and Stochastic Scheduling, Reidel.
58. Denardo, E.V. and Miller, B.L. (1968). An optimality criterion for discrete dynamic programming with no discounting, Ann. Math. Stat. 39, 1220-7.
59. Doob, J.L. (1955). A probability approach to the heat equation, Trans. Amer. Math. Soc. 80, 216-80.
60. Dumitriu, I., Tetali, P. and Winkler, P. (2003). On playing golf with two balls, SIAM J. Discret. Math. 16, 604-15.
61. Dunn, R.T. (2005). Bandit Problems on Parallel Machines, Ph.D. thesis, University of Edinburgh.
62. Dunn, R.T. and Glazebrook, K.D. (2001). The performance of index-based policies for bandit problems with stochastic machine availability, Adv. Appl. Prob. 33, 365-90.
63. Dunn, R.T. and Glazebrook, K.D. (2004). Discounted multiarmed bandit problems on a collection of machines with varying speeds, Math. Oper. Res. 29, 266-79.
64. Eick, S.G. (1988). Gittins procedures for bandits with delayed responses, J. R. Statist. Soc. B 50, 125-32.
65. El Karoui, N. and Karatzas, I. (1993). General Gittins index processes in discrete time, Proc. Natl. Acad. Sci. USA 90, 1232-6.
66. El Karoui, N. and Karatzas, I. (1994). Dynamic allocation problems in continuous time, Adv. Appl. Prob. 4, 255-86.
67. Eplett, W.J.R. (1986). Continuous-time allocation indices and their discrete-time approximations, Adv. Appl. Prob. 18, 724-46.
68. Fay, N.A. and Glazebrook, K.D. (1989). A general model for the scheduling of alternative stochastic jobs that may fail, Prob. Eng. Inform. Sci. 3, 199-221.
69. Fay, N.A. and Walrand, J.C. (1991). On approximately optimal index strategies for generalised arm problems, J. Appl. Prob. 28, 602-12.
70. Federgruen, A. and Groenvelt, H. (1988). Characterization and optimization of achieveable performances in general queueing systems, Oper. Res. 36, 733-41.
71. Ferguson, T.S. (1967). Mathematical Statistics: A Decision Theoretic Approach, Academic Press.
72. Filliger, R. and Hongler, M.-O. (2007). Explicit Gittins' indices for a class of superdiffusive processes, J. Appl. Prob. 44, 554-9.
73. French, S. (1982). Sequencing and Scheduling: An Introduction to the Mathematics of the Job-Shop, Ellis Norwood.
74. Frostig, E. and Weiss, G. (1999). Four proofs of Gittins' multi-armed bandit theorem, private communication.
75. Gal, S. (1980). Search Games, Academic Press.
76. Garbe, R. and Glazebrook, K.D. (1998a). Stochastic scheduling with priority classes, Math. Oper. Res. 23, 119-44.
77. Garbe, R. and Glazebrook, K.D. (1998b). Submodular returns and greedy heuristics for queueing scheduling problems, Oper. Res. 46, 336-46.
78. Garnett, O., Mandelbaum, A. and Reiman, M. (2002). Designing a call center with impatient customers, Manuf. Serv. Oper. Manage. 4, 208-27.
79. Gittins, J.C. (1979). Bandit processes and dynamic allocation indices (with discussion), J. R. Statist. Soc. B 41, 148-77.
80. Gittins, J.C. (1981). Multiserver scheduling of jobs with increasing completion rates, J. Appl. Prob. 18, 321-4.
81. Gittins, J.C. (1989). Multi-armed Bandit Allocation Indices, John Wiley & Sons, Inc.
82. Gittins, J.C. (2003). Stochastic models for the planning of pharmaceutical research, J. Statist. Theory Appl. 2, 198-214.
83. Gittins, J.C. and Jones, D.M. (1974a). A dynamic allocation index for the sequential design of experiments, in J. Gani, editor, Progress in Statistics, North-Holland, pages 241-66, read at the 1972 European Meeting of Statisticians, Budapest.
84. Gittins, J.C. and Jones, D.M. (1974b). A dynamic allocation index for new-product chemical research, Technical report, Cambridge University Engineering Department Report CUED/A -Mgt Stud/TR13, Cambridge University.
85. Gittins, J.C. and Jones, D.M. (1979). A dynamic allocation index for the discounted multi-armed bandit problem, Biometrika 66, 561-5.
86. Gittins, J.C. and Wang, Y.G. (1992). The learning component of dynamic allocation indices, Ann. Stat. 20, 1625-36.
87. Glazebrook, K.D. (1976). Stochastic scheduling with order constraints, Int. J. Sys. Sci. 7, 657-66.
88. Glazebrook, K.D. (1979). Stoppable families of alternative bandit processes, J. Appl. Prob. 16, 843-54.
89. Glazebrook, K.D. (1980a). On single-machine sequencing with order constraints, Naval Res. Logist. Quart. 27, 123-30.
90. Glazebrook, K.D. (1980b). On stochastic scheduling with precedence relations and switching costs, J. Appl. Prob. 17, 1016-24.
91. Glazebrook, K.D. (1981a). On non-preemptive strategies for stochastic scheduling problems in continuous time, Int. J. Sys. Sci. 12, 771-82.
92. Glazebrook, K.D. (1981b). On non-preemptive strategies in stochastic scheduling, Naval Res. Logist. Quart. 28, 289-300.
93. Glazebrook, K.D. (1982a). On the evaluation of fixed permutations as strategies in stochastic scheduling, Stoc. Proc. Appl., 13, 87.
94. Glazebrook, K.D. (1982b). On the evaluation of non-preemptive strategies in stochastic scheduling, in M.A.H. Dempster, J.K. Lenstra and A.H.G. Rinooy Kan, editors, Deterministic and Stochastic Scheduling, NATO Advanced Study Institute Series, Reidel, pages 375-84.
95. Glazebrook, K.D. (1982c). On the evaluation of suboptimal strategies for families of alternative bandit processes, J. Appl. Prob. 19, 716-22.
96. Glazebrook, K.D. (1983). Methods for the evaluation of permutations as strategies in stochastic scheduling, Manage. Sci. 29, 1142-55.
97. Glazebrook, K.D. (1991). On nonpreemptive policies for stochastic single-machine scheduling with breakdowns, Prob. Eng. Inform. Sci. 5, 77-87.
98. Glazebrook, K.D. and Boys, R.J. (1995). A class of Bayesian models for optimal exploration, J. R. Statist. Soc. B 57, 705-20.
99. Glazebrook, K.D. and Fay, N.A. (1987). On the scheduling of alternative stochastic jobs on a single machine, Adv. Appl. Prob. 19, 955-73.
100. Glazebrook, K.D. and Garbe, R. (1996). Reflections on a new approach to Gittins indexation, J. Oper. Res. Soc. 47, 1301-9.
101. Glazebrook, K.D. and Gittins, J.C. (1981). On single-machine scheduling with precedence relations and linear or discounted costs, Oper. Res. 29, 289-300.
102. Glazebrook, K.D., Hodge, D.J. and Kirkbride, C. (2010a). Monotone policies and indexability for bi-directional restless bandits, under review.
103. Glazebrook, K.D., Hodge, D.J. and Kirkbride, C. (2010b). General notions of indexability for queueing control and asset management, Ann. Appl. Prob. (in press).
104. Glazebrook, K.D., Kirkbride, C., Mitchell, H.M., Gaver, D.P. and Jacobs, P.A. (2007). Index policies for shooting problems, Oper. Res. 55, 769-81.
105. Glazebrook, K.D., Kirkbride, C. and Ouenniche, J. (2009). Index policies for the admission control and routing of impatient customers to heterogeneous service stations, Oper. Res. 57, 975-89.
106. Glazebrook, K.D., Kirkbride, C. and Ruiz-Hernandez, D. (2006). Spinning plates and squad systems: policies for bi-directional restless bandits, Adv. Appl. Prob. 38, 95-115.
107. Glazebrook, K.D., Lumley, R.R. and Ansell, P.S. (2003). Index heuristics for multiclass M/G/1 systems with nonpreemptive service and convex holding costs, Queueing Syst. 45, 81-111.
108. Glazebrook, K.D. and Niño-Mora, J. (2001). Parallel scheduling of multiclass M/M/m queues: approximate and heavy-traffic optimization of achievable performance, Oper. Res. 49, 609-23.
109. Glazebrook, K.D., Niño-Mora, J. and Ansell, P.S. (2002). Index policies for a class of discounted restless bandits, Adv. Appl. Prob. 34, 754-74.
110. Glazebrook, K.D. and Washburn, A. (2004). Shoot-look-shoot: a review and extension, Oper. Res. 52, 454-63.
111. Glazebrook, K.D. and Wilkinson, D.J. (2000). Index-based policies for discounted multiarmed bandits on parallel machines, Ann. Appl. Prob. 10, 877-96.
112. Guha, S. and Munagala, K. (2009). Multi-armed bandits with metric switching costs, in S. Albers, A. Marchetti-Spaccamela, Y. Matias, S. Nikoletseas and W. Thomas, editors, Automata, Languages and Programming, 36th Internatilonal Colloquium, ICALP 2009, Rhodes, Greece, July 5-12, 2009, proceedings, part II, Springer, pages 496-507.
113. Guha, S., Munagala, K. and Shi, P. (2009). Approximation algorithms for restless bandit problems, in C. Mathieu, editor, Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, Society for Industrial and Applied Mathematics, pages 28-37.
114. Harrison, J.M. (1975). Dynamic scheduling of a multiclass queue: discount optimality, Oper. Res. 23, 270-82.
115. Hauser, J.R., Urban, G.L., Liberali, G. and Braun, M. (2009). Website morphing, Market. Sci. 28, 202-23.
116. Henrici, P. (1964). Elements of Numerical Analysis, John Wiley & Sons, Inc.
117. Howard, R. (1960). Dynamic Programming and Markov Processes, MIT Press.
118. Howard, R. (1971). Dynamic Probabilistic Systems, volume II, Semi-Markov and Decision Processes, John Wiley & Sons, Inc.
119. Ishikida, T. and Varaiya, P. (1994).Multi-armed bandit problem revisited, J. Optim. Theory Appl., 83, 113-54.
120. Ishikida, T. and Wan, Y.W. (1997). Scheduling jobs that are subject to deterministic due dates and have deteriorating expected rewards, Prob. Eng. Inform. Sci., 11, 65-78.
121. Janson, S. and Peres, Y. (2001). Hitting times for random walks with restarts, Technical report, Department of Statistics, U.C. Berkeley, preprint.
122. Jones, D.M. (1970). A Sequential Method for Industrial Chemical Research, Master's thesis, U.C.W. Aberystwyth.
123. Jones, D.M. (1975). Search Procedures for Industrial Chemical Research, Ph.D. thesis, Cambridge.
124. Jones, M.L. and Koo, R. (2008). On the worth of perfect information in bandits with random discounting, Sequential Anal. 27, 58-67.
125. Jun, T. (2004). A survey on the bandit problem with switching costs, De Economist 152, 513-41.
126. Kadane, J.B. and Simon, H.A. (1977). Optimal strategies for a class of constrained sequential problems, Ann. Stat. 5, 237-55.
127. Kallenberg, L.C.M. (1986). A note on Katehakis and Y.-R. Chen's computation of the Gittins index, Math. Oper. Res. 11, 184-6.
128. Karatzas, I. (1984). Gittins indices in the dynamic allocation problem for diffusion processes, Adv. Appl. Prob. 12, 173-92.
129. Kaspi, H. and Mandelbaum, A. (1998). Multi-armed bandits in discrete and continuous time, Ann. Appl. Prob. 8, 270-90.
130. Katehakis, M.N. and Derman, C. (1986). Computing optimal sequential allocation rules in clinical trials, in J. Van Ryzin, editor, Adaptive Statistical Procedures and Related Topics, Institute of Mathematical Statistics, pages 29-39.
131. Katehakis, M.N. and Veinott, A.F. (1987). The multi-armed bandit problem: decomposition and computation, Math. Oper. Res. 12, 262-8.
132. Kelly, F.P. (1979). Contribution to the discussion of Gittins, J. R. Statist. Soc. B 41, 167-8.
133. Kelly, F.P. (1981). Multi-armed bandits with discount factor near one: the Bernoulli case, Ann. Stat. 9, 987-1001.
134. Kendall, D.G. (1951). Some problems in the theory of queues, J. R. Statist. Soc. B, 13, 85.
135. Kertz, R.P. (1986). Decision processes under total expected concomitant constraints with approximations to bandit processes, private communication.
136. Klimov, G.P. (1974). Time-sharing service systems I, Theory Probab. Appl. 19, 532-51.
137. Klimov, G.P. (1978). Time-sharing service systems II, Theory Probab. Appl. 23, 314-21.
138. Koopman, B.O. (1979). An operational critique of detection laws, Oper. Res., 27, 33.
139. Krishnamurthy, V. and Wahlberg, B. (2009). Partially observed Markov decision process multiarmed bandits -structural results, Math. Oper. Res. 34, 287-302.
140. Lai, T.L. and Robbins, H. (1985). Asymptotically efficient adaptive allocation rules, Adv. Appl. Math. 6, 4-22.
141. Lai, T.L. and Ying, Z. (1988). Open bandit processes and scheduling of queueing networks, Adv. Appl. Prob. 20, 447-72.
142. Lawler, E.L., Lenstra, J.K. and Rinnooy Kan, A.H.G. (1989). Theory of Sequencing & Scheduling, John Wiley & Sons, Ltd.
143. Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H.G. and Shmoys, D.B. (1993). Sequencing and scheduling: algorithms and complexity, in S.C. Graves, A.H.G. Rinnooy Kan and P.H. Zipkin, editors, Handbooks in Operations Research and Management Science, Elsevier, pages 445-522.
144. Lehnerdt, M. (1982). On the structure of discrete sequential search problems and of their solutions, Math Operationsforschung u. Statist., Ser. Optimization 13, 533-7.
145. Le Ny, J., Dahleh, M. and Feron, E. (2008). Multi-UAV dynamic routing with partial observations using restless bandit allocation indices, in American Control Conference, 2008, pages 4220-5.
146. Le Ny, J. and Feron, E. (2006). Restless bandits with switching costs: linear programming relaxations, performance bounds and limited lookahead policies, in Proceedings of the 2006 American Control Conference, pages 1587-92.
147. Levinthal, D. A. and Posen, H. E. (2011). Chasing a Moving Target: Learning in Dynamic Environments, Management Science. Available at SSRN: http://ssrn.com/abstract=1540100.
148. Lindley, D.V. (1960). Dynamic programming and decision theory, Appl. Statist. 10, 39-51.
149. Lippman, S.A. and McCall, J.J. (1981). The economics of belated information, Internat. Econ. Rev., 22, 46.
150. Liu, K. and Zhao, Q. (2010). Indexability of restless bandit problems and optimality of Whittle's index for dynamic multichannel access, IEEE T. Inf. Theor., preprint at http://arxiv.org/abs/0810.4658.
151. Mahajan, A. and Teneketzis, D. (2008). Multi-armed bandit problems, in A.O. Hero III, D.A. Castanon, D. Cochran and K. Kastella, editors, Foundations and Applications of Sensor Management, Springer, pages 121-51.
152. Mandelbaum, A. (1986). Discrete multi-armed bandits and multi-parameter processes, Prob. Theory Rel. Fields 71, 129-47.
153. Mandelbaum, A. (1988). Continuous multi-armed bandits and multi-parameter processes, Ann. Prob. 15, 1527-56.
154. Mannor, S. and Tsitsiklis, J.N. (2003). Lower bounds on the sample complexity of exploration in the multi-armed bandit problem, in B. Schölkopf and M.K. Warmuth, editors, Sixteenth Annual Conference on Computational Learning Theory, Springer, pages 418-32.
155. Manor, G. and Kress, M. (1997). Optimality of the greedy shooting strategy in the presence of incomplete damage information, Naval Res. Logist. 44, 613-22.
156. March, J.G. (1991). Exploration and exploitation in organisational learning, Organ. Sci. 2, 71-86.
157. Matula, D. (1964). A periodic optimal search, Amer. Math. Monthly 71, 15-21.
158. McCall, B.P. and McCall, J.J. (1981). Systematic search, belated information, and the Gittins index, Econ. Lett. 8, 327-33.
159. van Meighem, J.A. (1995). Dynamic scheduling with convex delay costs: the generalized cμ-rule, Adv. Appl. Prob. 5, 809-33.
160. Meilijson, I. and Weiss, G. (1977). Multiple feedback at a single-server station, Stoc. Proc. Appl. 5, 195-205.
161. Miller, R.A. (1984). Job matching and occupational choice, J. Political Econ. 92, 1086-120.
162. Nash, P. (1973). Optimal Allocation of Resources Between Research Projects, Ph.D. thesis, Cambridge.
163. Nash, P. (1980). A generalized bandit problem, J. R. Statist. Soc. B 42, 165-9.
164. Nash, P. and Gittins, J.C. (1977). A Hamiltonian approach to optimal stochastic resource allocation, Adv. Appl. Prob. 9, 55-68.
165. Nemhauser, G.L. and Wolsey, L.A. (1981). Maximizing submodular set functions: formulations and analysis of algorithms, in P. Hansen, editor, Annals of Discrete Mathematics 11, Studies on Graphs and Discrete Programming, North-Holland, pages 279-301.
166. Niño-Mora, J. (2001). Restless bandits, partial conservation laws and indexability, Adv. Appl. Prob. 33, 76-98.
167. Niño-Mora, J. (2007). A (2/3)n3 fast-pivoting algorithm for the Gittins index and optimal stopping of a Markov chain, INFORMS J. on Computing 19, 596-606.
168. Niño-Mora, J. (2011a). Klimov's model, in J.J. Cochran, editor, Wiley Encyclopedia of Operations Research and Management Science, John Wiley & Sons, Inc., pages 2265-74.
169. Niño-Mora, J. (2011b). Conservation laws and related applications, in J.J. Cochran, editor, Wiley Encyclopedia of Operations Research and Management Science, John Wiley & Sons, Inc., pages 939-50.
170. von Olivier, G. (1972). Cost-minimum priorities in queueing systems of type M/G/1, Elektron. Rechenanl. 14, 262-71.
171. van Oyen, M.P. and Teneketzis, D. (1994). Optimal stochastic scheduling of forest networks with switching penalties, Adv. Appl. Prob. 26, 474-97.
172. Pandelis, D.G. and Teneketzis, D. (1991). On the optimality of the Gittins index rule for multi-armed bandits with multiple plays, Math. Meth. Oper. Res. 50, 449-61.
173. Papadimitriou, C.H. and Steiglitz, K. (1982). Combinatorial Optimization: Algorithms and Complexity, Prentice-Hall.
174. Papadimitriou, C.H. and Tsitsiklis, J.N. (1999). The complexity of optimal queuing network control, Math. Oper. Res. 24, 293-305.
175. Peköz, E.A. (2003). Some memoryless bandit policies, J. Appl. Prob. 40, 250-6.
176. Pilnick, S.E., Glazebrook, K.D. and Gaver, D.P. (1991). Optimal replenishment of ships during combat, Naval Res. Logist., 637-88.
177. Pinedo, M.L. (2008). Scheduling: Theory, Algorithms and Systems, 3rd edition, Springer.
178. Presman, E.L. and Sonin, I.M. (1982). Sequential Control with Incomplete Information: The Bayesian Approach, Nauka.
179. Puterman, M.L. (2005). Markov Decision Processes: Discrete Stochastic Dynamic Programming, 2nd revised edition, John Wiley & Sons, Ltd.
180. Raiffa, H. and Schlaifer, R. (1961). Applied Statistical Decision Theory, Harvard Business School.
181. Rao, C.R. and Rubin, H. (1964). On a characterization of the Poisson distribution, Sankhya A 32, 265-70.
182. Righter, R. and Shanthikumar, G.J. (1998). Independently expiring multiarmed bandits, Prob. Eng. Inform. Sci. 12, 453-68.
183. Robbins, H. (1952). Some aspects of the sequential design of experiments, Bull. Amer. Math. Soc. 58, 527-35.
184. Robinson, D.R. (1982). Algorithms for evaluating the dynamic allocation index, Op. Res. Letters 1, 72-4.
185. Ross, K.W. (1989). Randomized and past-dependent policies for Markov decision processes with multiple constraints, Oper. Res. 37, 474-7.
186. Ross, S.M. (1970). Applied Probability Models with Optimization Applications, Holden-Day.
187. Ross, S.M. (1983). Introduction to Stochastic Dynamic Programming, Academic Press.
188. Rothkopf, M.H. (1966). Scheduling with random service times, Manage. Sci. 12, 707-13.
189. Sevcik, K.C. (1974). Scheduling for minimum total loss using service time distributions, JACM 21, 66-75.
190. Shanthikumar, J.G. and Yao, D.D. (1992). Multiclass queueing systems: polymatroidal structure and optimal scheduling control, Oper. Res., 40, S293-9.
191. Sidney, J.B. (1975). Decomposition algorithms for single-machine scheduling with precedence relations and deferral costs, Oper. Res. 23, 283-93.
192. Smith, W.E. (1956). Various optimizers for single-stage production, Naval Res. Logist. Quart., 1-2, 59-63.
193. Sonin, I.M. (2008). A generalized Gittins index for a Markov chain and its recursive calculation, Statist. Probab. Lett. 78, 1526-33.
194. Stone, L.D. (1975). Theory of Optimal Search, Academic Press.
195. Strümpfer, J. (1980). Search theory index, Technical Report TN-017-80, Institute for Maritime Technology, Simonstown, S. Africa.
196. Tcha, D.W. and Pliska, S.R. (1977). Optimal control of single-server queuing networks and multi-class M/G/1 queues with feedback, Oper. Res. 25, 248-58.
197. Teugels, J.L. (1976). A bibliography on semi-Markov processes, J. Comp. Appl. Maths., 2, 44.
198. Thron, C. (1984). The multi-armed bandit problem and optimality of the Gittins index strategy, private communication.
199. Tsitsiklis, J.N. (1986). A lemma on the multiarmed bandit problem, IEEE T. Automat. Contr. 31, 576-7.
200. Tsitsiklis, J.N. (1994). A short proof of the Gittins index theorem, Ann. Appl. Prob. 4, 194-9.
201. Tsoucas, P. (1991). The region of achieveable performance in a model of Klimov, Technical report, Research Report RC16543, IBM T.J. Watson Research Center, Yorktown Heights, NY.
202. Vanderbei, R. (2007). Linear Programming: Foundations and Extensions, Springer.
203. Varaiya, P., Walrand, J. and Buyukkoc, C. (1985). Extensions of the multiarmed bandit problem: the discounted case, IEEE T. Automat. Contr. 30, 426-39.
204. Veinott, A., Jr. (1966). On finding optimal policies in discrete dynamic programming with no discounting, Ann. Math. Stat. 37, 128-94.
205. Wald, A. (1950). Statistical Decision Functions, John Wiley & Sons, Inc.
206. Walrand, J. (1988). An Introduction to Queueing Networks, Prentice Hall.
207. Wang, Y.G. (1997). Error bounds for calculation of the Gittins indices, Austral. J. Statist. 39, 225-33.
208. Washburn, A. (2008). Application of multi-armed bandits to sensor management, in A.O. Hero III, D.A. Castanon, D. Cochran and K. Kastella, editors, Foundations and Applications of Sensor Management, Springer, pages 153-75.
209. Weber, R.R. (1980). Multi-server Stochastic Scheduling, Ph.D. thesis, Cambridge.
210. Weber, R.R. (1982). Scheduling jobs with stochastic processing requirement on parallel machines to minimise makespan or flowtime, J. Appl. Prob. 19, 167-82.
211. Weber, R.R. (1992). On the Gittins index for multiarmed bandits, Ann. Appl. Prob. 2, 1024-33.
212. Weber, R.R. (2007). Comments on: Dynamic priority allocation via restless bandit marginal productivity indices, TOP 15, 211-16.
213. Weber, R.R. and Weiss, G. (1990). On an index policy for restless bandits, J. Appl. Prob. 27, 637-48.
214. Weber, R.R. and Weiss, G. (1991). Addendum to 'On an index policy for restless bandits', Adv. Appl. Prob. 23, 429-30.
215. Weiss, G. (1988). Branching bandit processes, Prob. Eng. Inform. Sci. 2, 269-78.
216. Weitzman, M.L. (1979). Optimal search for the best alternative, Econometrica 47, 641-54.
217. Whittle, P. (1980). Multi-armed bandits and the Gittins index, J. R. Statist. Soc. B 42, 143-49.
218. Whittle, P. (1981). Arm-acquiring bandits, Ann. Prob. 9, 284-92.
219. Whittle, P. (1983). Optimization over Time, John Wiley & Sons, Ltd; volume 1, 1982, volume 2, 1983.
220. Whittle, P. (1988). Restless bandits: activity allocation in a changing world, in J. Gani, editor, A Celebration of Applied Probability, Applied Probability Special Volume 25A, Applied Probability Trust, pages 287-98.
221. Whittle, P. (1991). Book review: Multi-armed Bandit Allocation Indices, Stoch. Stoch. Rep. 35, 125-8.
222. Whittle, P. (1996). Optimal Control: Basics and Beyond, John Wiley & Sons, Ltd.
223. Whittle, P. (2005). Tax problems in the undiscounted case, J. Appl. Prob. 42, 754-65.
224. Whittle, P. (2007). Networks: Optimisation and Evolution, Cambridge University Press.
225. Yao, Y.C. (2006). Some results on the Gittins index for a normal reward process, in H. Ho, C. Ing and T. Lai, editors, Time Series and Related Topics: In Memory of Ching-Zong Wei, Institute of Mathematical Statistics, pages 284-94.