Adaptive crowdsourcing algorithms for the bandit survey problem

Journal of Machine Learning Research

Volumn 30, Issue , 2013, Pages 882-910

  Abraham, Ittai ^a   Alonso, Omar ^b   Kandylas, Vasilis ^b   Slivkins, Aleksandrs ^a  

^a MICROSOFT RESEARCH   (United States)

^b MICROSOFT   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

NATURAL LANGUAGE PROCESSING SYSTEMS; QUALITY CONTROL; SEARCH ENGINES; SURVEYS;

ADAPTIVE QUALITY; ANALYSIS AND SIMULATION; CROWDSOURCING PLATFORMS; HUMAN COMPUTATION; MULTI-ARMED BANDIT PROBLEM; NATURAL LANGUAGE PROCESSING; RELEVANCE EVALUATION; SIMPLE MODELING;

LEARNING ALGORITHMS;

EID: 84898026697     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (34)

1. Shipra Agrawal and Navin Goyal. Analysis of Thompson Sampling for the multi-armed bandit problem. In 25nd Conf. on Learning Theory (COLT), 2012.
2. Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3): 235-256, 2002a. Preliminary version in 15th ICML, 1998.
3. Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multi-armed bandit problem. SIAM J. Comput., 32(1): 48-77, 2002b. Preliminary version in 36th IEEE FOCS, 1995.
4. Moshe Babaioff, Shaddin Dughmi, Robert Kleinberg, and Aleksandrs Slivkins. Dynamic pricing with limited supply. In 13th ACM Conf. on Electronic Commerce (EC), 2012.
5. R. E. Bechhofer and D. Goldsman. Truncation of the bechhofer-kiefer-sobel sequential procedure for selecting the multinomial event which has the largest probability. Communications in Statistics Simulation and Computation, B14: 283-315, 1985.
6. R. E. Bechhofer, S. Elmaghraby, and N. Morse. A single-sample multiple decision procedure for selecting the multinomial event which has the highest probability. Annals of Mathematical Statistics, 30: 102119, 1959.
7. Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure Exploration in Multi-Armed Bandit Problems. Theoretical Computer Science, 412(19): 1832-1852, 2011. Preliminary version published in ALT 2009.
8. Chris Callison-Burch. Fast, cheap, and creative: Evaluating translation quality using amazon's mechanical turk. In ACL SIGDAT Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 286-295, 2009.
9. Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge Univ. Press, 2006.
10. Olivier Chapelle and Lihong Li. An Empirical Evaluation of Thompson Sampling. In 25th Advances in Neural Information Processing Systems (NIPS), 2011.
11. Xi Chen, Qihang Lin, and Dengyong Zhou. Optimistic knowledge gradient for optimal budget allocation in crowdsourcing. In 30th Intl. Conf. on Machine Learning (ICML), 2013.
12. Paul Dagum, Richard M. Karp, Michael Luby, and Sheldon M. Ross. An optimal algorithm for monte carlo estimation. SIAM J. on Computing, 29(5): 1484-1496, 2000.
13. Ofer Dekel and Ohad Shamir. Vox populi: Collecting high-quality labels from a crowd. In 22nd Conf. on Learning Theory (COLT), 2009.
14. Michael J. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. Crowddb: answering queries with crowdsourcing. In ACM SIGMOD Intl. Conf. on Management of Data (SIGMOD), pages 61-72, 2011.
15. Thore Graepel, Joaquin Quinonero Candela, Thomas Borchert, and Ralf Herbrich. Web-scale Bayesian click-through rate prediction for sponsored search advertising in Microsofts Bing search engine. In 27th Intl. Conf. on Machine Learning (ICML), pages 13-20, 2010.
16. Chien-Ju Ho and Jennifer Wortman Vaughan. Online task assignment in crowdsourcing markets. In 26th Conference on Artificial Intelligence (AAAI), 2012.
17. Chien-Ju H.o., Shahin Jabbari, and Jennifer Wortman Vaughan. Adaptive task assignment for crowd-sourced classification. In 30th Intl. Conf. on Machine Learning (ICML), 2013.
18. J. T. Ramey Jr. and K. Alam. A sequential procedure for selecting the most probable multinomial event. Biometrica, 66: 171-173, 1979.
19. Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In 11th Intl. Conf. on Autonomous Agents and Multiagent Systems (AAMAS), 2012.
20. Haim Kaplan, Eyal Kushilevitz, and Yishay Mansour. Learning with attribute costs. In 37th ACM Symp. on Theory of Computing (STOC), pages 356-365, 2005.
21. David R. Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In 25th Advances in Neural Information Processing Systems (NIPS), pages 1953-1961, 2011.
22. Robert Kleinberg. Nearly tight bounds for the continuum-armed bandit problem. In 18th Advances in Neural Information Processing Systems (NIPS), 2004.
23. Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-Armed Bandits in Metric Spaces. In 40th ACM Symp. on Theory of Computing (STOC), pages 681-690, 2008.
24. Tze Leung Lai and Herbert Robbins. Asymptotically efficient Adaptive Allocation Rules. Advances in Applied Mathematics, 6: 4-22, 1985.
25. Edith Law and Luis von Ahn. Human Computation. Morgan & Claypool Publishers, 2011.
26. Daniel J. Lizotte, Omid Madani, and Russell Greiner. Budgeted learning of naive-bayes classifiers. In 19th Conf. on Uncertainty in Artificial Intelligence (UAI), pages 378-385, 2003.
27. Steven L. Scott. A modern bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry, 26: 639658, 2010.
28. Omid Madani, Daniel J. Lizotte, and Russell Greiner. Active model selection. In 20th Conf. on Uncertainty in Artificial Intelligence (UAI), pages 357-365, 2004.
29. Shie Mannor and John N. Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. J. of Machine Learning Research (JMLR), 5: 623-648, 2004. Preliminary version in COLT, 2003.
30. Volodymyr Mnih, Csaba Szepesvári, and Jean-Yves Audibert. Empirical bernstein stopping. In 25th Intl. Conf. on Machine Learning (ICML), pages 672-679, 2008.
31. Victor S. Sheng, Foster J. Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In 14th ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (KDD), pages 614-622, 2008.
32. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In ACL SIGDAT Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 254-263, 2008.
33. William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4): 285-294, 1933.
34. Long Tran-Thanh, Matteo Venanzi, Alex Rogers, and Nicholas R. Jennings. Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks. In 12th Intl. Conf. on Autonomous Agents and Multiagent Systems (AAMAS), 2013.