Deterministic sequencing of exploration and exploitation for multi-armed bandit problems

IEEE Journal on Selected Topics in Signal Processing

Volumn 7, Issue 5, 2013, Pages 759-767

  Vakili, Sattar ^a   Liu, Keqin ^a   Zhao, Qing ^a  

^a Davis   (United States)

Author keywords

combinatorial multi armed bandit; decentralized multi armed bandit; deterministic sequencing of exploration and exploitation; Multi armed bandit; regret; restless multi armed bandit

Indexed keywords

DOMINATING SET PROBLEMS; EXPLORATION AND EXPLOITATION; EXPLORATION SEQUENCES; MULTI ARMED BANDIT; MULTI-ARMED BANDIT PROBLEM; REGRET; RESTLESS MULTI-ARMED BANDIT; SELECTION POLICIES;

ELECTRICAL ENGINEERING; SIGNAL PROCESSING;

OPTIMIZATION;

EID: 84884549238     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2013.2263494     Document Type: Article

References (28)

1. H. Robbins, "Some aspects of the sequential design of experiments," Bull. Amer. Math. Soc., vol. 58, no. 5, pp. 527-535, 1952.
2. T. Santner and A. Tamhane, Design of Experiments: Ranking and Selection. Boca Raton, FL, USA: CRC, 1984.
3. A. Mahajan and D. Teneketzis, , A. O. Hero, III, D. A. Castanon, D. Cochran, and K. Kastella, Eds., "Multi-armed bandit problems," in Foundations and Applications of Sensor Management. NewYork, NY, USA: Springer-Verlag, 2007.
4. T. Lai and H. Robbins, "Asymptotically efficient adaptive allocation rules," Adv. Appl. Math., vol. 6, no. 1, pp. 4-22, 1985.
5. R. Agrawal, "Sample mean based index policies with regret for the multi-armed bandit problem," Adv. Appl. Probab., vol. 27, pp. 1054-1078, 1995.
6. P. Auer, N. Cesa-Bianchi, and P. Fischer, "Finite-time analysis of the multiarmed bandit problem," Mach. Learn., vol. 47, pp. 235-256, 2002. (Pubitemid 34126111)
7. H. Liu, K. Liu, and Q. Zhao, "Learning in a changing world: Restless multi-armed bandit with unknown dynamics," IEEE Trans. Inf. Theory, vol. 59, no. 3, pp. 1902-1916, Mar. 2013.
8. K. Liu and Q. Zhao, "Adaptive shortest-path routing under unknown and stochastically varying link states," in Proc. 10th Int. Symp. Modeling Optimiz. Mobile, Ad Hoc, and Wireless Netw. (WiOpt), May 2012.
9. S. Bubeck, N. Cesa-Bianchi, and G. Lugosi, "Bandits With Heavy Tail," Sep. 2012 [Online]. Available: arXiv:1209.1727 [stat.ML]
10. K. Liu and Q. Zhao, "Multi-armed bandit problems with heavy tail reward distributions," in Proc. Allerton Conf. Commun., Control, Comput., Sep. 2011.
11. K. Liu and Q. Zhao, "Distributed learning in multi-armed bandit with multiple players," IEEE Trans. Signal Process., vol. 58, no. 11, pp. 5667-5681, Nov. 2010.
12. A. Anandkumar, N. Michael, A. K. Tang, and A. Swami, "Distributed algorithms for learning and cognitive medium access with logarithmic regret," IEEE J. Sel. Areas Commun., vol. 29, no. 4, pp. 731-745, Mar. 2011.
13. Y. Gai and B. Krishnamachari, "Decentralized online learning algorithms for opportunistic spectrum access," in Proc. IEEE Global Commun. Conf. (GLOBECOM '11), Houston, TX, USA, Dec. 2011.
14. C. Tekin and M. Liu, "Performance and convergence of multiuser online learning," in Proc. Int. Conf. Game Theory Netw. (GAMNETS), Apr. 2011.
15. D. Kalathil, N. Nayyar, and R. Jain, "Decentralized learning for multiplayer multi-armed bandits," in Proc. IEEE Conf. Decision Control (CDC), Dec. 2012, pp. 3960-3965.
16. D. Kalathil, N. Nayyar, and R. Jain, "Decentralized learning for multiplayer multi-armed bandits," IEEE Trans. Inf. Theory Apr. 2012 [Online]. Available: http://arxiv.org/abs/1206.3582, submitted for publication
17. C. Tekin and M. Liu, "Online learning of rested and restless bandits," IEEE Trans. Inf. Theory, vol. 58, no. 8, pp. 5588-5611, Aug. 2012.
18. W. Dai, Y. Gai, B. Krishnamachari, and Q. Zhao, "The non-Bayesian restless multi-armed bandit: A case of near-logarithmic regret," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), May 2011, pp. 2940-2943.
19. C. Tekin and M. Liu, "Approximately optimal adaptive learning in opportunistic spectrum access," in Proc. Int. Conf. Comput. Commun. (INFOCOM), Orlando, FL, USA, Mar. 2012.
20. C. Tekin and M. Liu, "Adaptive learning of uncontrolled restless bandits with logarithmic regret," in Proc. Allerton Conf. Commun. , Control, Comput., Sep. 2011.
21. Y. Gai, B. Krishnamachari, and R. Jain, "Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations," IEEE/ACM Trans. Netw., vol. 20, no. 5, pp. 1466-1478, Oct. 2012.
22. B. Awerbuch and R. Kleinberg, "Online linear optimization and adaptive routing," J. Comput. Syst. Sci., pp. 97-114, 2008. (Pubitemid 47625408)
23. Y. Gai, B. Krishnamachari, and M. Liu, "Online learning for combinatorial network optimization with restless Markovian rewards," in Proc. 9th Annu. IEEE Conf. Sens., Mesh, Ad Hoc Commun. Networks (SECON), 2012.
24. P. Chareka, O. Chareka, and S. Kennendy, "Locally sub-Gaussian random variable and the strong law of large numbers," Atlantic Electron. J. Math., vol. 1, no. 1, pp. 75-81, 2006.
25. R. Agrawal, "The continuum-armed bandit problem," SIAM J. Control Optimiz., vol. 33, no. 6, pp. 1926-1951, Nov. 1995.
26. W. Hoeffding, "Probability inequalities for sums of bounded random variables," J. Amer. Statist. Assoc., vol. 58, no. 301, pp. 13-30, Mar. 1963.
27. Y. Ren andH. Liang, "On the best constant in Marcinkiewicz-Zygmund inequality," Statist. Probab. Lett., vol. 53, pp. 227-233, Jun. 2001. (Pubitemid 33623382)
28. Y. Gai and B. Krishnamachari, "Decentralized online learning algorithms for opportunistic spectrum access," Tech. Rep., Mar. 2011 [Online]. Available: http://anrg.usc.edu/www/publications/papers/DMAB2011.pdf, Available at