Online learning of rested and restless bandits

IEEE Transactions on Information Theory

Volumn 58, Issue 8, 2012, Pages 5588-5611

  Tekin, Cem ^a   Liu, Mingyan ^a  

^a UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Exploration exploitation tradeoff; multiarmed bandits; online learning; opportunistic spectrum access (OSA); regret; restless bandits

Indexed keywords

EXPLORATION-EXPLOITATION TRADEOFF; MULTI ARMED BANDIT; ONLINE LEARNING; OPPORTUNISTIC SPECTRUM ACCESSES (OSA); REGRET; RESTLESS BANDITS;

EVOLUTIONARY ALGORITHMS; MARKOV PROCESSES;

E-LEARNING;

EID: 84863956678     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2012.2198613     Document Type: Article

References (26)

1. P. Auer, N. Cesa-Bianchi, Y. Freund, and R. Schapire, "The nonstochastic multiarmed bandit problem," SIAM J. Comput., vol. 32, pp. 48-77, Jan. 2002.
2. H. Robbins, "Some aspects of the sequential design of experiments," Bull. Amer. Math. Soc., vol. 55, pp. 527-535, 1952.
3. P. Auer, N. Cesa-Bianchi, and P. Fischer, "Finite-time analysis of the multiarmed bandit problem," Mach. Learn., vol. 47, pp. 235-256, 2002. (Pubitemid 34126111)
4. T. Lai and H. Robbins, "Asymptotically efficient adaptive allocation rules," Adv. Appl. Math., vol. 6, pp. 4-22, 1985.
5. V. Anantharam, P. Varaiya, and J. Walrand, "Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-Part I: I.I.D. rewards," IEEE Trans. Autom. Control, vol. 32, no. 11, pp. 968-976, Nov. 1987. (Pubitemid 18521625)
6. V. Anantharam, P. Varaiya, and J. Walrand, "Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-Part II: Markovian rewards," IEEE Trans. Autom. Control, vol. 32, no. 11, pp. 977-982, Nov. 1987. (Pubitemid 18521626)
7. R.Agrawal, "Samplemean based index policieswith regret for the multi-armed bandit problem," Adv. Appl. Probabil., vol. 27, no. 4, pp. 1054-1078, Dec. 1995.
8. A. Garivier and O. Cappe, "The KL-UCB algorithm for bounded stochastic bandits and beyond," in Proc. JMLR Workshop Conf., 2011, vol. 19, pp. 359-376.
9. V. Dani,T.P.Hayes, andS.M.Kakade, "Stochastic linear optimization under bandit feedback," in Proc. 21st Annu. Conf. Learn. Theory, Jul. 2008, pp. 355-366.
10. C. Tekin and M. Liu, "Online algorithms for the multi-armed bandit problem with Markovian rewards," in Proc. 48th Annu. Allerton Conf. Commun., Control, Comput., Sep. 2010, pp. 1675-1682.
11. C. Tekin and M. Liu, "Online learning in opportunistic spectrum access: A restless bandit approach," in Proc. 30th Annu. IEEE Int. Conf. Comput. Commun., Apr. 2011, pp. 2462-2470.
12. H. Liu, K. Liu, and Q. Zhao, Learning in a changing world: Non-Bayesian restless multi-armed bandit Univ. California Davis, Davis, 2010.
13. K. Liu and Q. Zhao, "Distributed learning in multi-armed bandit with multiple players," IEEE Trans. Signal Process., vol. 58, no. 11, pp. 5667-5681, Nov. 2010.
14. A. Anandkumar, N. Michael, A. Tang, and A. Swami, "Distributed algorithms for learning and cognitive medium access with logarithmic regret," IEEE J. Sel. Areas Commun., vol. 29, no. 4, pp. 731-745, Apr. 2011.
15. Y. Gai, B. Krishnamachari, and R. Jain, "Learning multiuser channel allocations in cognitive radio networks: A combinatorial multi-armed bandit formulation," in IEEE Symp. Dyn. Spectrum Access Netw. (DySPAN), Apr. 2010, pp. 1-9.
16. J. Gittins, "Bandit processes and dynamic allocation indices," J. Roy. Statist. Soc., vol. 41, no. 2, pp. 148-177, 1979.
17. P. Whittle, , J. Gani, Ed., "Restless bandits: Activity allocation in a changing world," in A Celebration of Applied Probability. Sheffield, U.K.: Applied Probability Trust, 1988, vol. 25A, pp. 287-298.
18. S. H. A. Ahmad, M. Liu, T. Javidi, Q. Zhao, and B. Krishnamachari, "Optimality of myopic sensing in multi-channel opportunistic access," IEEE Trans. Inf. Theory, vol. 55, no. 9, pp. 4040-4050, Sep. 2009.
19. P. Lezaud, "Chernoff-type bound for finite Markov chains," Ann. Appl. Probab., vol. 8, pp. 849-867, 1998.
20. J. Y. Audibert and R. M. Szepesvári, "Exploration- exploitation tradeoff using variance estimates in multi-armed bandits," Theoretical Comput. Sci., vol. 410, no. 19, pp. 1876-1902, 2009.
21. C. Tekin and M. Liu, "Adaptive learning of uncontrolled restless bandits with logarithmic regret," in Proc. 49th Annu. Allerton Conf. Commun., Control, Comput., Sep. 2011, pp. 983-990.
22. C. Papadimitriou and J. Tsitsiklis, "The complexity of optimal queuing network control," Math. Oper. Res., vol. 24, no. 2, pp. 293-305, May 1999.
23. W. Dai, Y. Gai, B. Krishnamachari, and Q. Zhao, "The non-Bayesian restless multi-armed bandit: A case of near-logarithmic regret," in Proc. Int. Conf. Acoust., Speech Signal Process., May 2011, pp. 2940-2943.
24. S. Guha, K. Mungala, and P. Shi, "Approximation algorithms for restless bandit problems," in Proc. 20th ACM-SIAM Symp. Discr. Algorithms, 2009, pp. 28-37.
25. C. Tekin and M. Liu, "Approximately optimal adaptive learning in opportunistic spectrum access," presented at the presented at the IEEE INFOCOM, Orlando, FL, Mar. 2012.
26. B. Eisenberg, "On the expectation of the maximum of i.i.d. geometric random variables," Statist. Probab. Lett., vol. 78, pp. 135-143, 2008.