Crowdsourcing label quality: a theoretical analysis

Science China Information Sciences

Volumn 58, Issue 11, 2015, Pages 1-12

  Wang, Wei ^a   Zhou, Zhi Hua ^a  

^a NANJING UNIVERSITY   (China)

Author keywords

crowdsourcing; label quality; majority voting; unlabeled data; worker selection

Indexed keywords

CROWDSOURCING;

CONSERVATIVE CONDITIONS; ERROR RATE; HIGH PROBABILITY; HUMAN MACHINE INTERACTION; LOW QUALITIES; MAJORITY VOTING; UNLABELED DATA; WORKER SELECTION;

QUALITY CONTROL;

EID: 84945286640     PISSN: 1674733X     EISSN: 18691919     Source Type: Journal    
DOI: 10.1007/s11432-015-5391-x     Document Type: Article

References (25)

1. Howe J. The rise of crowdsourcing. Wired, 2006, 14.06
2. Howe J. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. New York: Crown Publishing Group, 2008
3. Sheng V S, Provost F J, Ipeirotis P G. Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, 2008. 614–622
4. Snow R, O’Connor B, Jurafsky D, et al. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Honolulu, 2008. 254–263
5. Sorokin A, Forsyth D. Utility data annotation with Amazon Mechanical Turk. In: Proceedings of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop on Internet Vision, Anchorage, 2008. 1–8
6. Dempster A, Laird N, Rubin D. Maximum likelihood from incomplete data via the em algorithm. J Roy Stat Soc Ser B, 1977, 39: 1–38
7. Raykar V C, Yu S, Zhao L H, et al. Supervised learning from multiple experts: whom to trust when everyone lies a bit. In: Proceedings of the 26th International Conference on Machine Learning, Quebec, 2009
8. Raykar V C, Yu S, Zhao L H, et al. Learning from crowds. J Mach Learn Res, 2010, 11: 1297–1322
9. Whitehill J, Ruvolo P, Wu T, et al. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Bengio Y, Schuurmans D, Lafferty J D, et al., eds. Advances in Neural Information Processing Systems 22. Cambridge: MIT Press, 2009. 2035–2043
10. Welinder P, Branson S, Belongie S, et al. The multidimensional wisdom of crowds. In: Lafferty J D, Williams C K I, Shawe-Taylor J, et al., eds. Advances in Neural Information Processing Systems 23. Cambridge: MIT Press, 2010. 2424–2432
11. Raykar V C, Yu S. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J Mach Learn Res, 2012, 13: 491–518
12. Liu Q, Peng J, Ihler A T. Variational inference for crowdsourcing. In: Bartlett P L, Pereira F C N, Burges C J C, et al., eds. Advances in Neural Information Processing Systems 25. Cambridge: MIT Press, 2012. 701–709
13. Zhou D, Platt J C, Basu S, et al. Learning from the wisdom of crowds by minimax entropy. In: Bartlett P L, Pereira F C N, Burges C J C, et al., eds. Advances in Neural Information Processing Systems 25. Cambridge: MIT Press, 2012. 2204–2212
14. Yan Y, Rosales R, Fung G, et al. Active learning from crowds. In: Proceedings of the 28th International Conference on Machine Learning, Bellevue, 2011. 1161–1168
15. Wauthier F L, Jordan M I. Bayesian bias mitigation for crowdsourcing. In: Shawe-Taylor J, Zemel R S, Bartlett P L, et al., eds. Advances in Neural Information Processing Systems 24. Cambridge: MIT Press, 2011. 1800–1808
16. Karger D R, Oh S, Shah D. Iterative learning for reliable crowdsourcing systems. In: Shawe-Taylor J, Zemel R S, Bartlett P L, et al., eds. Advances in Neural Information Processing Systems 24. Cambridge: MIT Press, 2011. 1953–1961
17. Auer P, Cesa-Bianchi N, Freund Y, et al. The nonstochastic multiarmed bandit problem. SIAM J Comput, 2003, 32: 48–77
18. Ho C-J, Vaughan JW. Online task assignment in crowdsourcing markets. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence, Toronto, 2012
19. Buchbinder N, Naor J. Online primal-dual algorithms for covering and packing problems. In: Proceedings of the 13th Annual European Symposium, Palma de Mallorca, 2005. 689–701
20. Ho C-J, Jabbari S, Vaughan J W. Adaptive task assignment for crowdsourced classification. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, 2013. 534–542
21. Chen X, Lin Q, Zhou D. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, 2013. 64–72
22. Dekel O, Shamir O. Vox populi: Collecting high-quality labels from a crowd. In: Proceedings of the 22nd Conference on Learning Theory, Quebec, 2009
23. Tran-Thanh L, Stein S, Rogers A, et al. Efficient crowdsourcing of unknown experts using multi-armed bandits. In: Proceedings of the 20th European Conference on Artificial Intelligence, Montpellier, 2012. 768–773
24. Abraham I, Alonso O, Kandylas V, et al. Adaptive crowdsourcing algorithms for the bandit survey problem. In: Proceedings of the 26th Annual Conference on Learning Theory, Princeton, 2013. 882–910
25. Tsybakov A. Optimal aggregation of classifiers in statistical learning. Ann Stat, 2004, 32: 135–166