Ranked bandits in metric spaces: Learning diverse rankings over large document collections

Journal of Machine Learning Research

Volumn 14, Issue 1, 2013, Pages 399-436

  Slivkins, Aleksandrs ^a   Radlinski, Filip ^b   Gollapudi, Sreenivas ^a  

^a MICROSOFT RESEARCH   (United States)

^b MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Clickthrough data; Contextual bandits; Diversity; Metric spaces; Multi armed bandits; Online learning; Regret

Indexed keywords

CLICKTHROUGH DATA; CONTEXTUAL BANDITS; DIVERSITY; METRIC SPACES; MULTI ARMED BANDIT; ONLINE LEARNING; REGRET;

OPTIMIZATION; SET THEORY; TOPOLOGY;

ALGORITHMS;

EID: 84875138796     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (52)

1. Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In 21th Conf. on Learning Theory (COLT), pages 263-274, 2008.
2. Rajeev Agrawal. The continuum-armed bandit problem. SIAM J. Control and Optimization, 33(6):1926-1951, 1995.
3. David J. Aldous. Exchangeability and related topics. In Ecole d'Été de Probabilités de Saint-Flour XIII, pages 1-198, 1985.
4. Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. J. of Machine Learning Research (JMLR), 3:397-422, 2002. Preliminary version in 41st IEEE FOCS, 2000.
5. Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002a. Preliminary version in 15th ICML, 1998. (Pubitemid 34126111)
6. Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput., 32(1):48-77, 2002b. Preliminary version in 36th IEEE FOCS, 1995.
7. Peter Auer, Ronald Ortner, and Csaba Szepesvári. Improved rates for the stochastic continuumarmed bandit problem. In 20th Conf. on Learning Theory (COLT), pages 454-468, 2007.
8. Baruch Awerbuch and Robert Kleinberg. Online linear optimization and adaptive routing. J. of Computer and System Sciences, 74(1):97-114, February 2008. Preliminary version in 36th ACM STOC, 2004. (Pubitemid 47625408)
9. Yair Bartal. Probabilistic approximations of metric spaces and its algorithmic applications. In IEEE Symp. on Foundations of Computer Science (FOCS), 1996.
10. Dirk Bergemann and Juuso Välimäki. Bandit problems. In Steven Durlauf and Larry Blume, editors, The New Palgrave Dictionary of Economics, 2nd ed. Macmillan Press, 2006.
11. Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multiarmed bandit problems. Foundations and Trends in Machine Learning (Draft under submission), 2012. Available at www.princeton. edu/?sbubeck/pub.html.
12. Sébastien Bubeck and Rémi Munos. Open loop optimistic planning. In 23rd Conf. on Learning Theory (COLT), pages 477-489, 2010.
13. Sébastien Bubeck, Rémi Munos, Gilles Stoltz, and Csaba Szepesvari. Online optimization in xarmed bandits. J. of Machine Learning Research (JMLR), 12:1587-1627, 2011. Preliminary version in NIPS 2008.
14. Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. Learning to rank using gradient descent. In Intl. Conf. on Machine Learning (ICML), pages 89-96, 2005. (Pubitemid 43183320)
15. Jaime G. Carbonell and Jade Goldstein. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In ACM Intl. Conf. on Research and Development in Information Retrieval (SIGIR), pages 335-336, 1998.
16. Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games. Cambridge Univ. Press, 2006.
17. Wei Chu and Zoubin Ghahramani. Gaussian processes for ordinal regression. J. of Machine Learning Research, 6:1019-1041, 2005.
18. Wei Chu, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandits with linear payoff functions. In 14th Intl. Conf. on Artificial Intelligence and Statistics (AISTATS), 2011.
19. Varsha Dani, Thomas P. Hayes, and Sham Kakade. The price of bandit information for online optimization. In 20th Advances in Neural Information Processing Systems (NIPS), 2007.
20. Jittat Fakcharoenphol, Satish Rao, and Kunal Talwar. A tight bound on approximating arbitrary metrics by tree metrics. J. of Computer and System Sciences, 69(3):485-497, 2004.
21. Abraham Flaxman, Adam Kalai, and H. Brendan McMahan. Online convex optimization in the bandit setting: Gradient descent without a gradient. In 16th ACM-SIAM Symp. on Discrete Algorithms (SODA), pages 385-394, 2005. (Pubitemid 40851394)
22. Daniel Golovin, Andreas Krause, and Matthew Streeter. Online learning of assignments. In Advances in Neural Information Processing Systems (NIPS), 2009.
23. Anupam Gupta, Robert Krauthgamer, and James R. Lee. Bounded geometries, fractals, and low-distortion embeddings. In IEEE Symp. on Foundations of Computer Science (FOCS), 2003.
24. Elad Hazan and Satyen Kale. Better algorithms for benign bandits. In 20th ACM-SIAM Symp. on Discrete Algorithms (SODA), pages 38-47, 2009.
25. Elad Hazan and Nimrod Megiddo. Online learning with prior information. In 20th Conf. on Learning Theory (COLT), pages 499-513, 2007.
26. Thorsten Joachims. Optimizing search engines using clickthrough data. In 8th ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (KDD), pages 133-142, 2002.
27. Satyen Kale, Lev Reyzin, and Robert E. Schapire. Non-stochastic bandit slate problems. In 24th Advances in Neural Information Processing Systems (NIPS), pages 1054-1062, 2010.
28. Robert Kleinberg. Nearly tight bounds for the continuum-armed bandit problem. In 18th Advances in Neural Information Processing Systems (NIPS), 2004.
29. Robert Kleinberg and Aleksandrs Slivkins. Sharp dichotomies for regret minimization in metric spaces. In 21st ACM-SIAM Symp. on Discrete Algorithms (SODA), 2010.
30. Robert Kleinberg, Alexandru Niculescu-Mizil, and Yogeshwer Sharma. Regret bounds for sleeping experts and bandits. In 21st Conf. on Learning Theory (COLT), pages 425-436, 2008a.
31. Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in metric spaces. In 40th ACM Symp. on Theory of Computing (STOC), pages 681-690, 2008b.
32. Levente Kocsis and Csaba Szepesvari. Bandit based Monte-Carlo planning. In 17th European Conf. on Machine Learning (ECML), pages 282-293, 2006. (Pubitemid 44618839)
33. Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6:4-22, 1985.
34. John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. In 21st Advances in Neural Information Processing Systems (NIPS), 2007.
35. Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In 19th Intl. World Wide Web Conf. (WWW), 2010.
36. Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextualbandit-based news article recommendation algorithms. In 4th ACM Intl. Conf. on Web Search and Data Mining (WSDM), 2011.
37. Tyler Lu, DáSvid Pál, and Martin Pál. Showing relevant ads via Lipschitz context multi-armed bandits. In 14th Intl. Conf. on Artificial Intelligence and Statistics (AISTATS), 2010.
38. Odalric-Ambrym Maillard and Rémi Munos. Online learning in adversarial lipschitz environments. In European Conf. on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), pages 305-320, 2010.
39. Rémi Munos and Pierre-Arnaud Coquelin. Bandit algorithms for tree search. In 23rd Conf. on Uncertainty in Artificial Intelligence (UAI), 2007.
40. Sandeep Pandey, Deepak Agarwal, Deepayan Chakrabarti, and Vanja Josifovski. Bandits for taxonomies: A model-based approach. In SIAM Intl. Conf. on Data Mining (SDM), 2007.
41. Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. Learning diverse rankings with multiarmed bandits. In 25th Intl. Conf. on Machine Learning (ICML), pages 784-791, 2008.
42. Philippe Rigollet and Assaf Zeevi. Nonparametric bandits with covariates. In 23rd Conf. on Learning Theory (COLT), pages 54-66, 2010.
43. Aleksandrs Slivkins. Contextual bandits with similarity information. http://arxiv.org/abs/0907.3986, 2009. Has been published in 24th COLT 2011.
44. Aleksandrs Slivkins. Multi-armed bandits on implicit metric spaces. In 25th Advances in Neural Information Processing Systems (NIPS), 2011.
45. Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In 27th Intl. Conf. on Machine Learning (ICML), pages 1015-1022, 2010.
46. Matthew Streeter and Daniel Golovin. An online algorithm for maximizing submodular functions. In Advances in Neural Information Processing Systems (NIPS), pages 1577-1584, 2008.
47. Rangarajan K. Sundaram. Generalized bandit problems. In David Austen-Smith and John Duggan, editors, Social Choice and Strategic Decisions: Essays in Honor of Jeffrey S. Banks (Studies in Choice and Welfare), pages 131-162. Springer, 2005. First appeared as Working Paper, Stern School of Business, 2003.
48. Michael J. Taylor, John Guiver, Stephen Robertson, and Tom Minka. Softrank: Optimizing nonsmooth rank metrics. In ACM Intl. Conf. on Web Search and Data Mining (WSDM), pages 77-86, 2008.
49. Taishi Uchiya, Atsuyoshi Nakamura, and Mineichi Kudo. Algorithms for adversarial bandit problems with multiple plays. In 21st Intl. Conf. on Algorithmic Learning Theory (ALT), pages 375-389, 2010.
50. Chih-Chun Wang, Sanjeev R. Kulkarni, and H. Vincent Poor. Bandit problems with side observations. IEEE Trans. on Automatic Control, 50(3):338355, 2005.
51. Yizao Wang, Jean-Yves Audibert, and Rémi Munos. Algorithms for infinitely many-armed bandits. In Advances in Neural Information Processing Systems (NIPS), pages 1729-1736, 2008.
52. Michael Woodroofe. A one-armed bandit problem with a concomitant variable. J. Amer. Statist. Assoc., 74(368), 1979.