How many workers to ask? Adaptive exploration for collecting high quality labels

SIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2016, Pages 473-482

  Abraham, Ittai ^a   Alonso, Omar ^b   Kandylas, Vasilis ^b   Patel, Rajesh ^b   Shelford, Steven ^b   Slivkins, Aleksandrs ^b  

^a VMWARE INC   (United States)

^b MICROSOFT   (United States)

Author keywords

Adaptive algorithms; Assessments; Crowdsourcing; Ground truth; Label quality; Multi armed bandits

Indexed keywords

ADAPTIVE ALGORITHMS; CROWDSOURCING; INFORMATION RETRIEVAL;

ADAPTIVE EXPLORATIONS; ASSESSMENTS; AUTOMATIC CREATIONS; CROWDSOURCING PLATFORMS; GROUND TRUTH; HISTORICAL PERFORMANCE; MULTI ARMED BANDIT; PERFORMANCE QUALITY;

QUALITY CONTROL;

EID: 84980317079     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2911451.2911514     Document Type: Conference Paper

References (27)

1. Ittai Abraham, Omar Alonso, Vasilis Kandylas, and Aleksandrs Slivkins. Adaptive crowdsourcing algorithms for the bandit survey problem. In 26th COLT, 2013.
2. Omar Alonso and Stefano Mizzaro. Using crowdsourcing for TREC relevance assessment. Inf. Process. Manage., 48(6):1053-1066, 2012.
3. Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002.
4. R. E. Bechhofer, S. Elmaghraby, and N. Morse. A single-sample multiple decision procedure for selecting the multinomial event which has the highest probability. Annals of Mathematical Statistics, 30:102-119, 1959.
5. R. E. Bechhofer and D. Goldsman. Truncation of the bechhofer-kiefer-sobel sequential procedure for selecting the multinomial event which has the largest probability. Communications in Statistics - Simulation and Computation, B14:283-315, 1985.
6. Roi Blanco, Harry Halpin, Daniel M. Herzig, Peter Mika, Jeffrey Pound, Henry S. Thompson, and Duc Thanh Tran. Repeatable and reliable search system evaluation using crowdsourcing. In 34th SIGIR, 2011.
7. Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems. Foundations and Trends in Machine Learning, 5(1):1-122, 2012.
8. Chris Callison-Burch. Fast, cheap, and creative: Evaluating translation quality using amazon's mechanical turk. In EMNLP, pages 286-295, 2009.
9. Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge Univ. Press, 2006.
10. Xi Chen, Qihang Lin, and Dengyong Zhou. Optimistic knowledge gradient for optimal budget allocation in crowdsourcing. In 30th ICML, 2013.
11. Paul Dagum, Richard M. Karp, Michael Luby, and Sheldon M. Ross. An optimal algorithm for monte carlo estimation. SIAM J. on Computing, 29(5):1484-1496, 2000.
12. Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics, pages 20-28, 1979.
13. Ofer Dekel and Ohad Shamir. Vox populi: Collecting high-quality labels from a crowd. In 22nd COLT, 2009.
14. J. C. Gittins. Bandit processes and dynamic allocation indices (with discussion). J. Roy. Statist. Soc. Ser. B, 41:148-177, 1979.
15. Chien-Ju H.o., Shahin Jabbari, and Jennifer Wortman Vaughan. Adaptive task assignment for crowdsourced classification. In 30th ICML, 2013.
16. Panagiotis Ipeirotis, Foster Provost, and Jing Wang. Quality Management on Amazon Mechanical Turk. In HCOMP, 2010.
17. J. T. Ramey Jr. and K. Alam. A sequential procedure for selecting the most probable multinomial event. Biometrica, 66:171-173, 1979.
18. Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In 11th AAMAS, 2012.
19. David R. Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In 25th NIPS, pages 1953-1961, 2011.
20. Volodymyr Mnih, Csaba Szepesvári, and Jean-Yves Audibert. Empirical bernstein stopping. In 25th ICML, pages 672-679, 2008.
21. David Oleson, Alexander Sorokin, Greg Laughlin, Vaughn Hester, John Le, and Lukas Biewald. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. In Human Computation Workshop, 2011.
22. Aditya G. Parameswaran, Hector Garcia-Molina, Hyunjung Park, Neoklis Polyzotis, Aditya Ramesh, and Jennifer Widom. Crowdscreen: algorithms for filtering data with humans. In ACM SIGMOD, pages 361-372, 2012.
23. Falk Scholer, Andrew Turpin, and Mark Sanderson. Quantifying test collection quality based on the consistency of relevance judgements. In Proceeding of the 34th International ACM SIGIR, pages 1063-1072, 2011.
24. Victor S. Sheng, Foster J. Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In 14th KDD, 2008.
25. Aleksandrs Slivkins and Jennifer Wortman Vaughan. Online decision making in crowdsourcing markets: Theoretical challenges. SIGecom Exchanges, 12(2), December 2013.
26. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP, pages 254-263, 2008.
27. William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.